dspot icon indicating copy to clipboard operation
dspot copied to clipboard

Published 20 hours ago • verified publisher icon STAMP-project

study the optimal configuration of amplifiers

Open monperrus opened this issue 5 years ago • 13 comments

We should have a default optimal configuration of amplifiers, and maybe a set of 2/3 alternative ones.

Zeming Dong will work on this.

monperrus avatar Apr 30 '19 08:04 monperrus

A represents the I-Amplification; P represents the Project.

P1=javapoet; P2=mybatis-3; P3=traccar; P4=stream-lib; P5=mustache.java; P6=twilio-java; P7=jsoup; P8=protostuff; P9=logback; P10=retrofit.

Figure in brackets denotes the Incresae Killed.

  • A1=MethodAdd [Average Score (1.08%) ] P1 (0.0%): FieldSpaceTest (322-322-0.0%), NameAllocatorTest (322-322-0.0%), ParameterSpecTest (322-322-0.0%), TypeNameTest (321-322-0.3%) P2 (0.0%): MetaClassTest (105-107-1.9%) P3 (0.0%): ObdDecoderTest (58-58-0%), At2000ProtocolDecoderTest (560-560-0%), MiscFormatterTest(7-7-0%) P4 (0.0%): TestMurmurHash (20-20-0.0%), TestDoublyLinkedList (16-16-0.0%), TestLookup3Hash (20-20-0.0%), TestICardinality (134-134-0.0%) P6 (5.2%): RequestTest (21-26-23.8%), PrefixedCollapsibleMapTest(12-12-0.0%), AllTimeTest(32-32-0.0%), DailyTest (32-32-0.0%) P7 (0.2%): AttributeTest (330-332-0.6%), CharacterReaderTest (359-359-0.0%), TokenQueueTest (362-362-0.0%), AttributesTest (330-331-0.3%) P8 (0.%): TailDelimiterTest(345-345-0.0%), LinkBufferTest(345-345-0.0%), CodedDataInputTest(345-345-0.0%), CodedInputTest(345-349-1.2%) P10 (0.0%): CallTest (125-125-0.0%), RequestBuilderAndroidTest,(125-125-0.0%), CallAdapterTest (125-125-0.0%), ExecutorCallAdapterFactoryTest (124-124-0.0%)

  • A2=MethodRemove [Average Score (1.10%) ] P1 (0.0%): FieldSpaceTest (322-322-0.0%), NameAllocatorTest (322-322-0.0%), ParameterSpecTest (322-322-0.0%), TypeNameTest (321-322-0.3%) P2 (0.0%): MetaClassTest (105-107-1.9%) P3 (0.16%): ObdDecoderTest (58-58-0%), At2000ProtocolDecoderTest (560-561-0.2%), MiscFormatterTest(7-7-0%) P4 (0.0%): TestMurmurHash (20-20-0.0%), TestDoublyLinkedList (16-16-0.0%), TestLookup3Hash (20-20-0.0%), TestICardinality (134-134-0.0%) P6 (5.2%): RequestTest (21-26-23.8%), PrefixedCollapsibleMapTest(12-12-0.0%), AllTimeTest(32-32-0.0%), DailyTest (32-32-0.0%) P7 (0.15%): AttributeTest(330-331-0.3%), CharacterReaderTest (359-359-0.0%), TokenQueueTest (359-359-0.0%), AttributesTest (330-331-0.3%) P8 (0.%): TailDelimiterTest(345-345-0.0%), LinkBufferTest(345-345-0.0%), CodedDataInputTest(345-345-0.0%), CodedInputTest(345-349-1.2%) P10 (0.0%)

  • A3=TestDataMutator [Average Score (1.12%) ] P1 (0.0%): FieldSpaceTest (322-322-0.0%), NameAllocatorTest (322-322-0.0%), ParameterSpecTest (322-322-0.0%), TypeNameTest (321-322-0.3%) P2 (0.0%): MetaClassTest (105-107-1.9%) P3 (0.0%): ObdDecoderTest (58-58-0%), At2000ProtocolDecoderTest (560-560-0%), MiscFormatterTest(7-7-0%) P4 (0.0%): TestMurmurHash (20-20-0.0%), TestDoublyLinkedList (16-16-0.0%), TestLookup3Hash (20-20-0.0%), TestICardinality (134-134-0.0%) P6 (5.2%): RequestTest (21-26-23.8%), PrefixedCollapsibleMapTest(12-12-0.0%), AllTimeTest(32-32-0.0%), DailyTest (32-32-0.0%) P7 (0.38%): AttributeTest(330-334-1.2%), CharacterReaderTest (361-361-0.0%), TokenQueueTest (361-361-0.0%), AttributesTest (330-331-0.3%) P8 (0.%): TailDelimiterTest(345-345-0.0%), LinkBufferTest(345-345-0.0%), CodedInputTest(345-349-1.2%) P10 (0.0%)

  • A4=MethodGeneratorAmplifier [Average Score (1.28%) ] P1 (0.0%): FieldSpaceTest (322-322-0.0%), NameAllocatorTest (322-322-0.0%), ParameterSpecTest (322-322-0.0%), TypeNameTest (321-325-1.2%) P2 (0.0%): MetaClassTest (105-107-1.9%) P3 (0.0%): ObdDecoderTest (58-58-0%), At2000ProtocolDecoderTest (560-560-0%), MiscFormatterTest(7-7-0%) P4 (0.0%): TestMurmurHash (20-20-0.0%), TestDoublyLinkedList (16-16-0.0%), TestLookup3Hash (20-20-0.0%), TestICardinality (134-134-0.0%) P6 (5.2%): RequestTest (21-26-23.8%), PrefixedCollapsibleMapTest(12-12-0.0%), AllTimeTest(32-32-0.0%), DailyTest (32-32-0.0%) P7 (1.20%): AttributeTest(330-340-3.0%), CharacterReaderTest (359-359-0.0%), TokenQueueTest (359-360-0.3%), AttributesTest (330-335-1.5%) P8 (0.%): TailDelimiterTest(345-345-0.0%), LinkBufferTest(345-350-1.4%), CodedDataInputTest(345-345-0.0%), CodedInputTest(345-359-4.1%) P10 (0.0%)

  • A5=ReturnValueAmplifier [Average Score (1.15%) ] P1 (0.0%): FieldSpaceTest (322-322-0.0%), NameAllocatorTest (322-322-0.0%), ParameterSpecTest (322-322-0.0%), TypeNameTest (321-323-0.6%) P2 (0.0%): MetaClassTest (105-107-1.9%) P3 (0.0%): ObdDecoderTest (58-58-0%), At2000ProtocolDecoderTest (560-560-0%), MiscFormatterTest(7-7-0%) P4 (0.0%): TestMurmurHash (20-20-0.0%), TestDoublyLinkedList (16-16-0.0%), TestLookup3Hash (20-20-0.0%), TestICardinality (134-134-0.0%) P6 (5.2%): RequestTest (21-26-23.8%), PrefixedCollapsibleMapTest(12-12-0.0%), AllTimeTest(32-32-0.0%), DailyTest (32-32-0.0%) P7 (0.53%): AttributeTest(330-334-1.2%), CharacterReaderTest (359-359-0.0%), TokenQueueTest (359-359-0.0%), AttributesTest (330-333-0.9%) P8 (0.%): TailDelimiterTest(345-345-0.0%), LinkBufferTest(345-350-1.4%), CodedDataInputTest(345-345-0.0%), CodedInputTest(345-347-0.6%) P10 (0.0%)

  • A6=StringLiteralAmplifier [Average Score (1.09%) ] P1 (0.0%): FieldSpaceTest (322-322-0.0%), NameAllocatorTest (322-322-0.0%), ParameterSpecTest (322-322-0.0%), TypeNameTest (321-322-0.3%) P2 (0.0%): MetaClassTest (105-107-1.9%) P3 (0.0%): ObdDecoderTest (58-58-0%), At2000ProtocolDecoderTest (560-560-0%), MiscFormatterTest(7-7-0%) P4 (0.0%): TestMurmurHash (20-20-0.0%), TestDoublyLinkedList (16-16-0.0%), TestLookup3Hash (20-20-0.0%), TestICardinality (134-134-0.0%) P6 (5.2%): RequestTest (21-26-23.8%), PrefixedCollapsibleMapTest(12-12-0.0%), AllTimeTest(32-32-0.0%), DailyTest (32-32-0.0%) P7 (0.23%): AttributeTest(330-332-0.6%), CharacterReaderTest (359-359-0.0%), TokenQueueTest (359-359-0.0%), AttributesTest (330-331-0.3%) P8 (0.%): TailDelimiterTest(345-345-0.0%), LinkBufferTest(345-345-0.0%), CodedInputTest(345-347-0.6%) P10 (0.0%)

  • A7=NumberLiteralAmplifier [Average Score (1.09%) ] P1 (0.0%): FieldSpaceTest (322-322-0.0%), NameAllocatorTest (322-322-0.0%), ParameterSpecTest (322-322-0.0%), TypeNameTest (321-322-0.3%) P2 (0.0%): MetaClassTest (105-107-1.9%) P3 (0.0%): ObdDecoderTest (58-58-0%), At2000ProtocolDecoderTest (560-560-0%), MiscFormatterTest(7-7-0%) P4 (0.0%): TestMurmurHash (20-20-0.0%), TestDoublyLinkedList (16-16-0.0%), TestLookup3Hash (20-20-0.0%), TestICardinality (134-134-0.0%) P6 (5.2%): RequestTest (21-26-23.8%), PrefixedCollapsibleMapTest(12-12-0.0%), AllTimeTest(32-32-0.0%), DailyTest (32-32-0.0%) P7 (0.23%): AttributeTest(330-332-0.6%), CharacterReaderTest (359-359-0.0%), TokenQueueTest (359-359-0.0%), AttributesTest (330-331-0.3%) P8 (0.%): TailDelimiterTest(345-345-0.0%), LinkBufferTest(345-345-0.0%), CodedInputTest(345-350-1.4%) P10 (0.0%)

  • A8=BooleanLiteralAmplifier [Average Score (1.07%) ] P1 (0.0%): FieldSpaceTest (322-322-0.0%), NameAllocatorTest (322-322-0.0%), ParameterSpecTest (322-322-0.0%), TypeNameTest (321-322-0.3%) P2 (0.0%): MetaClassTest (105-107-1.9%) P3 (0.0%): ObdDecoderTest (58-58-0%), At2000ProtocolDecoderTest (560-560-0%), MiscFormatterTest(7-7-0%) P4 (0.0%): TestMurmurHash (20-20-0.0%), TestDoublyLinkedList (16-16-0.0%), TestLookup3Hash (20-20-0.0%), TestICardinality (134-134-0.0%) P6 (5.2%): RequestTest (21-26-23.8%), PrefixedCollapsibleMapTest(12-12-0.0%), AllTimeTest(32-32-0.0%), DailyTest (32-32-0.0%) P7 (0.15%): AttributeTest(330-331-0.3%), CharacterReaderTest (359-359-0.0%), TokenQueueTest (361-361-0.0%), AttributesTest (330-331-0.3%) P8 (0.%): TailDelimiterTest(345-345-0.0%), LinkBufferTest(345-345-0.0%), CodedInputTest(345-347-0.6%) P10 (0.0%)

  • A9=CharLiteralAmplifier [Average Score (1.07%) ] P1 (0.0%): FieldSpaceTest (322-322-0.0%), NameAllocatorTest (322-322-0.0%), ParameterSpecTest (322-322-0.0%), TypeNameTest (321-322-0.3%) P2 (0.0%): MetaClassTest (105-107-1.9%) P3 (0.0%): ObdDecoderTest (58-58-0%), At2000ProtocolDecoderTest (560-560-0%), MiscFormatterTest(7-7-0%) P4 (0.0%): TestMurmurHash (20-20-0.0%), TestDoublyLinkedList (16-16-0.0%), TestLookup3Hash (20-20-0.0%), TestICardinality (134-134-0.0%) P6 (5.2%): RequestTest (21-26-23.8%), PrefixedCollapsibleMapTest(12-12-0.0%), AllTimeTest(32-32-0.0%), DailyTest (32-32-0.0%) P7 (0.15%): AttributeTest(329-330-0.3%), CharacterReaderTest (361-361-0.0%), TokenQueueTest (359-359-0.0%), AttributesTest (329-330-0.3%) P8 (0.%): TailDelimiterTest(345-345-0.0%), LinkBufferTest(345-345-0.0%), CodedInputTest(345-347-0.6%) P10 (0.0%)

  • A10=AllLiteralAmplifiers[Average Score (1.09%) ] P1 (0.0%): FieldSpaceTest (322-322-0.0%), NameAllocatorTest (322-322-0.0%), ParameterSpecTest (322-322-0.0%), TypeNameTest (321-322-0.3%) P2 (0.0%): MetaClassTest (105-107-1.9%) P3 (0.0%): ObdDecoderTest (58-58-0%), At2000ProtocolDecoderTest (560-560-0%), MiscFormatterTest(7-7-0%) P4 (0.0%): TestMurmurHash (20-20-0.0%), TestDoublyLinkedList (16-16-0.0%), TestLookup3Hash (20-20-0.0%), TestICardinality (135-135-0.0%) P6 (5.2%): RequestTest (21-26-23.8%), PrefixedCollapsibleMapTest(12-12-0.0%), AllTimeTest(32-32-0.0%), DailyTest (32-32-0.0%) P7 (0.23%): AttributeTest(330-332-0.6%), CharacterReaderTest (361-361-0.0%), TokenQueueTest (359-359-0.0%), AttributesTest (330-331-0.3%) P8 (0.%): TailDelimiterTest(345-345-0.0%), LinkBufferTest(345-345-0.0%), CodedInputTest(345-350-1.4%) P10 (0.0%): CallTest (125-125-0.0%), RequestBuilderAndroidTest,(125-125-0.0%), CallAdapterTest (125-125-0.0%), ExecutorCallAdapterFactoryTest (124-124-0.0%)

  • A11=NullifierAmplifier [Average Score (1.07%) ] P1 (0.0%): FieldSpaceTest (322-322-0.0%), NameAllocatorTest (322-322-0.0%), ParameterSpecTest (322-322-0.0%),, TypeNameTest (321-322-0.3%) P2 (0.0%): MetaClassTest (105-107-1.9%) P3 (0.0%): ObdDecoderTest (58-58-0%), At2000ProtocolDecoderTest (560-560-0%), MiscFormatterTest(7-7-0%) P4 (0.0%): TestMurmurHash (20-20-0.0%), TestDoublyLinkedList (16-16-0.0%), TestLookup3Hash (20-20-0.0%), TestICardinality (134-134-0.0%) P6 (5.2%): RequestTest (21-26-23.8%), PrefixedCollapsibleMapTest(12-12-0.0%), AllTimeTest(32-32-0.0%), DailyTest (32-32-0.0%) P7 (0.15%): AttributeTest(330-331-0.3%), CharacterReaderTest (360-360-0.0%), TokenQueueTest (359-359-0.0%), AttributesTest (330-331-0.3%) P8 (0.%): TailDelimiterTest(345-345-0.0%), LinkBufferTest(345-345-0.0%), CodedInputTest(345-347-0.6%) P10 (0.0%)

zemingd avatar Jun 07 '19 13:06 zemingd

Classify amplifier methods into

  1. Method calls: MethodAdd, MethodRemove, MethodGeneratorAmplifier
  2. Literals: StringLiteralAmplifier, NumberLiteralAmplifier, BooleanLiteralAmplifier, CharLiteralAmplifier, AllLiteralAmplifiers, TestDataMutator
  3. Test objects: ReturnValueAmplifier, NullifierAmplifier

Jsoup.AttributeTest

  • Auto-correlation

  1. Method calls： MethodAdd+MethodRemove, (330-332-0.6%) MethodAdd+MethodGenerator, (330-341-3.3%) MethodRemove+MethodGenerator(330-342-3.6%), MethodAdd+MethodRemove+MethodGenerator（330-339-2.7%）

  2. Literals： StringLiteralAmplifier+NumberLiteralAmplifier（330-332-0.6%） StringLiteralAmplifier+BooleanLiteralAmplifier（330-332-0.6%） StringLiteralAmplifier+CharLiteralAmplifier（330-332-0.6%） NumberLiteralAmplifier+BooleanLiteralAmplifier（330-331-0.3%） NumberLiteralAmplifier+CharLiteralAmplifier（330-332-0.6%） BooleanLiteralAmplifier+CharLiteralAmplifier（330-331-0.3%） StringLiteralAmplifier+NumberLiteralAmplifier+BooleanLiteralAmplifier（330-332-0.6%） StringLiteralAmplifier+NumberLiteralAmplifier+CharLiteralAmplifier（330-332-0.6%） NumberLiteralAmplifier+BooleanLiteralAmplifier+CharLiteralAmplifier (330-331-0.3%) StringLiteralAmplifier+BooleanLiteralAmplifier+CharLiteralAmplifier (330-332-0.6%) TestDataMutator+StringLiteralAmplifier+NumberLiteralAmplifier（330-333-0.9%） TestDataMutator+StringLiteralAmplifier+BooleanLiteralAmplifier（330-333-0.9%） TestDataMutator+StringLiteralAmplifier+CharLiteralAmplifier（330-333-0.9%） TestDataMutator+NumberLiteralAmplifier+BooleanLiteralAmplifier（330-333-0.9%） TestDataMutator+NumberLiteralAmplifier+CharLiteralAmplifier（330-333-0.9%） TestDataMutator+BooleanLiteralAmplifier+CharLiteralAmplifier（330-333-0.9%） TestDataMutator+StringLiteralAmplifier+NumberLiteralAmplifier+BooleanLiteralAmplifier（330-333-0.9%） TestDataMutator+StringLiteralAmplifier+NumberLiteralAmplifier+CharLiteralAmplifier（330-333-0.9%） TestDataMutator+NumberLiteralAmplifier+BooleanLiteralAmplifier+CharLiteralAmplifier（330-333-0.9%） TestDataMutator+StringLiteralAmplifier+BooleanLiteralAmplifier+CharLiteralAmplifier（330-333-0.9%） TestDataMutator+AllLiteralAmplifiers（330-333-0.9%） TestDataMutator+StringLiteralAmplifier（330-333-0.9%） TestDataMutator+NumberLiteralAmplifier（330-333-0.9%） TestDataMutator+BooleanLiteralAmplifier（330-333-0.9%） TestDataMutator+CharLiteralAmplifier（330-333-0.9%）

  3. Test objects： ReturnValueAmplifier+NullifierAmplifier（330-331-0.3%）

  • Cross-correlation

  1. Method calls+Literals TestDataMutator+MethodAdd+MethodRemove（330-333-0.9%） TestDataMutator+MethodAdd+MethodGenerator（330-333-0.9%） TestDataMutator+MethodRemove+MethodGenerator（330-333-0.9%） TestDataMutator+MethodAdd+MethodRemove+MethodGenerator（330-334-1.2%） AllLiteralAmplifiers+MethodAdd+MethodRemove（330-333-0.9%） AllLiteralAmplifiers+MethodAdd+MethodGenerator（330-333-0.9%） AllLiteralAmplifiers+MethodRemove+MethodGenerator（330-333-0.9%） AllLiteralAmplifiers+MethodAdd+MethodRemove+MethodGenerator（330-334-1.2%） MethodGenerator+NumberLiteralAmplifier（330-331-0.3%） MethodGenerator+StringLiteralAmplifier（330-332-0.6%） MethodGenerator+BooleanLiteralAmplifier（330-331-0.3%） MethodGenerator+CharLiteralAmplifier（330-331-0.3%）

  2. Method calls+Test objects MethodRemove+MethodGenerator+ReturnValueAmplifier+NullifierAmplifier（330-331-0.3%） MethodRemove+MethodGenerator+ReturnValueAmplifier（330-331-0.3%） MethodRemove+MethodGenerator+NullifierAmplifier（330-331-0.3%） MethodGenerator+ReturnValueAmplifier（330-331-0.3%） MethodRemove+ReturnValueAmplifier（330-331-0.3%） MethodGenerator+NullifierAmplifier（330-331-0.3%） MethodRemove+NullifierAmplifier（330-331-0.3%）

  3. Literals+Test objects AllLiteralAmplifiers+ReturnValueAmplifier+NullifierAmplifier（330-332-0.6%） AllLiteralAmplifiers+ReturnValueAmplifier（330-332-0.6%） AllLiteralAmplifiers+NullifierAmplifier（330-332-0.6%） TestDataMutator+ReturnValueAmplifier+NullifierAmplifier（330-333-0.9%） TestDataMutator+ReturnValueAmplifier（330-333-0.9%） TestDataMutator+NullifierAmplifier（330-333-0.9%）

  4. Method calls+Literals+Test objects MethodRemove+MethodGenerator+TestDataMutator+ReturnValueAmplifier（330-333-0.9%） MethodRemove+MethodGenerator+TestDataMutator+NullifierAmplifier（330-333-0.9%） MethodRemove+TestDataMutator+ReturnValueAmplifier（330-333-0.9%） MethodRemove+TestDataMutator+NullifierAmplifier（330-333-0.9%） TestDataMutator+MethodAdd+MethodRemove+MethodGenerator+ReturnValueAmplifier（330-334-1.2%） TestDataMutator+MethodAdd+MethodRemove+MethodGenerator+NullifierAmplifier（330-334-1.2%） TestDataMutator+MethodAdd+MethodRemove+MethodGenerator+NullifierAmplifier+ReturnValueAmplifier (330-333-0.9%)

zemingd avatar Jul 09 '19 12:07 zemingd

Could you add the absolute numbers, eg before:100 killed, after 110 killed (100-110-10%)?

monperrus avatar Jul 09 '19 13:07 monperrus

We have 11 configurations, so we will have the results for:

  • 11 single amplifier runs
  • 11*11 double amplifier runs
  • 11*11*11 double amplifier runs
  • etc.
  • (note that AllLiteralAmplifiers is already a combination by default)

monperrus avatar Jul 09 '19 13:07 monperrus

Hello,

TestDataMutator is the old implementation of AllLiteralAmplifiers, so it should be in the category Literals, right?

Thank you.

danglotb avatar Jul 09 '19 13:07 danglotb

Hi Benjamin, Do you mean that TestDataMutatoris legacy code that can be discarded? even removed from the code base?

monperrus avatar Jul 09 '19 13:07 monperrus

@Deeplus Auto-correlation: do you mean within-group combination, eg all amplifiers coming from the same group? Cross-correlation: do you mean inter-group combination, eg at least two amplifiers coming from the same group?

monperrus avatar Jul 09 '19 13:07 monperrus

The fact is that since TestDataMutator is less stronger, i.e. it generates less amplified test methods at each iteration, it allows DSpot to obtain better result because good amplified test methods are not discarded. (You know that when we have a lot of amplified test methods, we discard a part of them to have a reasonable number of amplified test methods to pursue the amplification)

So it happens, that you can obtain better result using TestDataMutator instead of AllLiteralAmplifiers.

This why I kept it in DSpot.

This is an open question: according to our resources, how much can we explore?

It would be awesome to fine-tune the input amplification, at the level of the amplifier, in order to not exceed this arbitrary number of test methods and not waste any resource.

danglotb avatar Jul 09 '19 13:07 danglotb

I see. What about renaming TestDataMutator? for instance into FastLiteralAmplifier?

monperrus avatar Jul 09 '19 13:07 monperrus

according to our resources, how much can we explore?

That's indeed one of the hardest questions that @Deeplus is answering.

monperrus avatar Jul 09 '19 13:07 monperrus

Hi all,

  1. @monperrus Auto-correlation means within-group combination; Cross-correlation means inter-group combination.

  2. @danglotb ReturnValueAmplifier and NullifierAmplifier belong to the category Test objects, is this right?

zemingd avatar Jul 09 '19 14:07 zemingd

@danglotb What about renaming TestDataMutator whose name is confusing? for instance into FastLiteralAmplifier? Another idea?

monperrus avatar Jul 18 '19 12:07 monperrus

Hi, @Deeplus Yeah this is the correct category.

@monperrus okay to modify the name of the amplifier TestDataMutator to FastLiteralAmplifier

danglotb avatar Jul 18 '19 13:07 danglotb