singletons
singletons copied to clipboard
goldfirere
Split out `singletons-base` test suite into its own package
Before I explain why I want to do this, let me first make some observations about singletons-base's minimum GHC requirements.
Historically speaking, singletons-base has always been tied to a particular GHC version, as its implementation almost always requires the latest version of GHC to build. Or, at the very least, to build without needing heavy amounts of CPP. While this has been true of every singletons release that I can recall, GHC 9.4 is, remarkably, not that different from 9.2 in terms of its impact on singletons-base. In fact, one can compile the current state of the master branch with both 9.2 and 9.4, requiring no other changes than some minor changes to version bounds.
This is pretty remarkable, and I think it would be worth considering making the next release of singletons-base build with both GHC 9.2 and 9.4, given how straightforward it would be to do so. There's only one sticking point with this plan that I am aware of: the singletons-base test suite. The output of this test suite is intimately tied to one particular GHC version, and indeed, the output of the test suite changes depending on whether you use 9.2 or 9.4. Granted, it's mostly whitespace and formatting changes (e.g., rendering Type instead of Data.Kind.Type), but these are changes nonetheless.
I think it would be a shame to have the test suite be the sole reason why we can't widen the support window for singletons-base a little. In light of this, I propose that we split out the singletons-base test suite into its own singletons-base-tests package. The GHC support window for singletons-base-tests would be even tighter than it would be for singletons-base. We could continue to run the test suite in CI, and we could even upload singletons-base-tests to Hackage so that those who want to run it themselves can continue to do so.
Aside from the GHC support window benefits, another benefit to doing this is that singletons-base would no longer require a custom Setup.hs script; only singletons-base-tests would. Custom Setup.hs scripts often pose issues for things like cross-compilation, so this seems like a natural way to remove that roadblock.
I don't yet find this plan compelling. Instead, would it be possible just to have two (or more) different sets of golden files, keyed on the GHC version? That seems like it has the same amount of duplication as your plan (still need to maintain multiple golden files), but less user-facing complexity.
This might be different if simplifying Setup.hs was a key motivator, but that is presented more like a "hey, look, extra bonus!" than a reason to go down this route in the first place.
My main motivator is making it easier to support multiple versions of GHC as is reasonable. I disagree that maintaining multiple sets of golden files would involve the same amount of duplication. In fact, it would involve substantially more duplication, since now we have to double the amount of files we keep track of in version control. (There are a lot of golden files.) I'm not excited about that prospect.
Ah -- I think I had missed the point that the testsuite would cover only some GHC versions, so there would be no duplication. But then I think we can achieve that goal without a separate package: just conditionally have the testsuite fail on the wrong GHC version. (Or maybe succeed vacuously?) In any case, it worries me a bit that this plan would mean that we advertise that singletons works with GHC-x, but it would be completely untested there.
I think we can achieve that goal without a separate package: just conditionally have the testsuite fail on the wrong GHC version. (Or maybe succeed vacuously?)
Ah, I hadn't considered that. Indeed, making the test suite conditional sounds like substantially less work, so I like that idea better.
In any case, it worries me a bit that this plan would mean that we advertise that
singletonsworks with GHC-x, but it would be completely untested there.
I think "completely untested" is a bit of an exaggeration. After all, most of the interesting work that singletons does is at compile time, so being able to compile singletons-base at all is already providing quite a bit of test coverage. It's certainly not giving you all of the test coverage that the test suite provides, but it's quite a bit more than nothing.
Relatedly, I've never been happy with making all of the test suites be golden tests, as it's coupled far too tightly with the output of GHC's -ddump-splices flag. I would favor moving some of these tests over to simple unit tests, at which point you could actually run them on multiple versions of GHC without remorse.
I agree about making the tests more well targeted. And I also agree about "completely untested"..... but the testsuite does exist for a reason. Maybe the tested-with bit in a .cabal file is sufficient to label this fact, though.
To make this a bit more concrete, 47 out of the 122 tests in the singletons-base test suite fail with GHC 9.4:
Test suite singletons-base-test-suite: RUNNING...
Testsuite
Singletons
Nat: FAIL (5.34s)
198c198
< data SNat :: Nat -> GHC.Types.Type
---
> data SNat :: Nat -> Type
215c215
< -> GHC.Types.Type) t1) t2)
---
> -> Type) t1) t2)
233c233
< -> GHC.Types.Type) t1) t2) t3)
---
> -> Type) t1) t2) t3)
266c266
< -> GHC.Types.Type) t1) t2)
---
> -> Type) t1) t2)
301,302c301
< Data.Type.Equality.TestEquality (SNat :: Nat
< -> GHC.Types.Type) where
---
> Data.Type.Equality.TestEquality (SNat :: Nat -<truncated>
Use --accept or increase --size-cutoff to see full output.
Use -p '$0=="Testsuite.Singletons.Nat"' to rerun this test only.
Empty: FAIL (3.52s)
5c5
< data SEmpty :: Empty -> GHC.Types.Type
---
> data SEmpty :: Empty -> Type
Use -p '$0=="Testsuite.Singletons.Empty"' to rerun this test only.
Maybe: FAIL (5.07s)
94c94
< data SMaybe :: forall a. Maybe a -> GHC.Types.Type
---
> data SMaybe :: forall a. Maybe a -> Type
111c111
< -> GHC.Types.Type) t1) t2)
---
> -> Type) t1) t2)
129c129
< -> GHC.Types.Type) t1) t2) t3)
---
> -> Type) t1) t2) t3)
167,168c167
< Data.Type.Equality.TestEquality (SMaybe :: Maybe a
< -> GHC.Types.Type) where
---
> Data.Type.Equality.TestEquality (SMaybe :: Maybe a -> Type) where
172,173c171
< Data.Type.Coercion.TestCoercion (SMaybe :: Maybe a
< -> GHC.Types<truncated>
Use --accept or increase --size-cutoff to see full output.
Use -p '/Maybe/' to rerun this test only.
BoxUnBox: FAIL (3.64s)
41c41
< data SBox :: forall a. Box a -> GHC.Types.Type
---
> data SBox :: forall a. Box a -> Type
Use -p '/BoxUnBox/' to rerun this test only.
Operators: SKIP
Use -p '/Operators/' to rerun this test only.
HigherOrder: SKIP
Use -p '/HigherOrder/' to rerun this test only.
Contains: OK (4.99s)
AsPattern: SKIP
Use -p '/AsPattern/' to rerun this test only.
DataValues: SKIP
Use -p '/DataValues/' to rerun this test only.
EqInstances: SKIP
Use -p '/EqInstances/' to rerun this test only.
CaseExpressions: OK (5.08s)
Star: SKIP
Use -p '/Star/' to rerun this test only.
ReturnFunc: SKIP
Use -p '/ReturnFunc/' to rerun this test only.
Lambdas: FAIL (5.42s)
755c755
< data SFoo :: forall a b. Foo a b -> GHC.Types.Type
---
> data SFoo :: forall a b. Foo a b -> Type
Use -p '$0=="Testsuite.Singletons.Lambdas"' to rerun this test only.
LambdasComprehensive: SKIP
Use -p '/LambdasComprehensive/' to rerun this test only.
Error: OK (4.98s)
TopLevelPatterns: FAIL (5.17s)
35c35
< data SBool :: Bool -> GHC.Types.Type
---
> data SBool :: Bool -> Type
46c46
< data SFoo :: Foo -> GHC.Types.Type
---
> data SFoo :: Foo -> Type
Use -p '/TopLevelPatterns/' to rerun this test only.
LetStatements: SKIP
Use -p '/LetStatements/' to rerun this test only.
LambdaCase: OK (5.26s)
Sections: SKIP
Use -p '/Sections/' to rerun this test only.
PatternMatching: SKIP
Use -p '/PatternMatching/' to rerun this test only.
Records: FAIL (5.12s)
64c64
< data SRecord :: forall a. Record a -> GHC.Types.Type
---
> data SRecord :: forall a. Record a -> Type
Use -p '/Records/' to rerun this test only.
T29: OK (4.95s)
T33: FAIL (4.68s)
30c30
< Lazy pattern converted into regular pattern in promotion
---
> Lazy pattern converted into regular pattern during singleton generation.
36c36
< Lazy pattern converted into regular pattern during singleton generation.
---
> Lazy pattern converted into regular pattern in promotion
Use -p '$0=="Testsuite.Singletons.T33"' to rerun this test only.
T54: OK (4.68s)
Classes: SKIP
Use -p '$0=="Testsuite.Singletons.Classes"' to rerun this test only.
Classes2: SKIP
Use -p '/Classes2/' to rerun this test only.
FunDeps: OK (4.55s)
T78: OK (4.63s)
OrdDeriving: FAIL (6.61s)
466c466
< data SNat :: Nat -> GHC.Types.Type
---
> data SNat :: Nat -> Type
478c478
< data SFoo :: forall a b c d. Foo a b c d -> GHC.Types.Type
---
> data SFoo :: forall a b c d. Foo a b c d -> Type
481,485c481,485
< (Sing n)
< -> (Sing n)
< -> (Sing n)
< -> (Sing n)
< -> SFoo (A n n n n :: Foo a b c d)
---
> (Sing n) ->
> (Sing n) ->
> (Sing n) ->
> (Sing n) ->
> SFoo (A n n n n :: Foo a b c d)
487,491c487,491
< (Sing n)
< -> (Sing n)
< -> (Sing n)
< -> (Sing n)
< -> SFoo (B n n n n :: Foo a b c d)
---
> (Sing n) ->
> (Sing n) ->
> (Sing n) ->
> (Sing n) ->
> SFoo (B n n n n :: Foo a b c d)
493,497c493,497
< (Sing n)
< -> (Sing n)
< -> (Sin<truncated>
Use --accept or increase --size-cutoff to see full output.
Use -p '/OrdDeriving/' to rerun this test only.
BoundedDeriving: OK (4.81s)
BadBoundedDeriving: OK (2.54s)
EnumDeriving: FAIL (4.77s)
73c73
< data SFoo :: Foo -> GHC.Types.Type
---
> data SFoo :: Foo -> Type
87c87
< data SQuux :: Quux -> GHC.Types.Type
---
> data SQuux :: Quux -> Type
102c102
< -> GHC.Types.Type) t)
---
> -> Type) t)
106c106
< -> GHC.Types.Type) t)
---
> -> Type) t)
195c195
< -> GHC.Types.Type) t)
---
> -> Type) t)
199c199
< <truncated>
Use --accept or increase --size-cutoff to see full output.
Use -p '/Singletons.EnumDeriving/' to rerun this test only.
BadEnumDeriving: OK (2.69s)
Fixity: OK (4.48s)
Undef: OK (4.50s)
T124: OK (4.33s)
T136: FAIL (4.36s)
118,119c118
< -> Sing (Apply (SuccSym0 :: TyFun [Bool] [Bool]
< -> GHC.Types.Type) t)
---
> -> Sing (Apply (SuccSym0 :: TyFun [Bool] [Bool] -> Type) t)
122,123c121
< -> Sing (Apply (PredSym0 :: TyFun [Bool] [Bool]
< -> GHC.Types.Type) t)
---
> -> Sing (Apply (PredSym0 :: TyFun [Bool] [Bool] -> Type) t)
127c125
< -> GHC.Types.Type) t)
---
> -> Type) t)
131c129
< -> GHC.Types.Type) t)
---
> -> Type) t)
Use -p '$0=="Testsuite.Singletons.T136"' to rerun this test only.
T136b: FAIL (3.36s)
52c52
< -> Sing (Apply (MethSym0 :: TyFun Bool Bool -> GHC.Types.Type) t)
---
> -> Sing (Apply (MethSym0 :: TyFun Bool Bool -> Type) t)
Use -p '/T136b/' to rerun this test only.
T153: OK (1.76s)
T157: OK (1.78s)
T159: FAIL (3.27s)
22,23c22,23
< type ST0 :: T0 -> GHC.Types.Type
< data ST0 :: T0 -> GHC.Types.Type
---
> type ST0 :: T0 -> Type
> data ST0 :: T0 -> Type
105,106c105,106
< type ST1 :: T1 -> GHC.Types.Type
< data ST1 :: T1 -> GHC.Types.Type
---
> type ST1 :: T1 -> Type
> data ST1 :: T1 -> Type
218c218
< data ST2 :: T2 -> GHC.Types.Type
---
> data ST2 :: T2 -> Type
Use -p '/T159/' to rerun this test only.
T167: OK (4.42s)
T145: OK (3.38s)
PolyKinds: OK (3.32s)
PolyKindsApp: OK (3.09s)
T150: SKIP
Use -p '/T150/' to rerun this test only.
T160: OK (3.86s)
T163: FAIL (3.30s)
27c27
< data (%+) :: forall a b. (+) a b -> GHC.Types.Type
---
> data (%+) :: forall a b. (+) a b -> Type
Use -p '/T163/' to rerun this test only.
T166: OK (2.76s)
T172: OK (4.10s)
T175: OK (3.34s)
T176: OK (4.48s)
T178: FAIL (4.52s)
145c145
< data SOcc :: Occ -> GHC.Types.Type
---
> data SOcc :: Occ -> Type
164c164
< -> GHC.Types.Type) t1) t2)
---
> -> Type) t1) t2)
179c179
< -> GHC.Types.Type) t1) t2)
---
> -> Type) t1) t2)
213c213
< -> GHC.Types.Type) t1) t2) t3)
---
> -> Type) t1) t2) t3)
249c249
< -> GHC.Types.Type) where
---
> -> Type) where
253c253
< <truncated>
Use --accept or increase --size-cutoff to see full output.
Use -p '/T178/' to rerun this test only.
T183: OK (4.77s)
T184: OK (4.80s)
T187: FAIL (4.18s)
64c64
< data SEmpty :: Empty -> GHC.Types.Type
---
> data SEmpty :: Empty -> Type
75c75
< -> GHC.Types.Type) t1) t2)
---
> -> Type) t1) t2)
82c82
< -> GHC.Types.Type) t1) t2)
---
> -> Type) t1) t2)
87c87
< -> GHC.Types.Type) where
---
> -> Type) where
91c91
< -> GHC.Types.Type) where
---
> -> Type) where
Use -p '/T187/' to rerun this test only.
T190: FAIL (4.55s)
156c156
< data ST :: T -> GHC.Types.Type where ST :: ST (T :: T)
---
> data ST :: T -> Type where ST :: ST (T :: T)
167c167
< -> GHC.Types.Type) t1) t2)
---
> -> Type) t1) t2)
174c174
< -> GHC.Types.Type) t1) t2)
---
> -> Type) t1) t2)
186c186
< -> GHC.Types.Type) t)
---
> -> Type) t)
190c190
< -> GHC.Types.Type) t)
---
> <truncated>
Use --accept or increase --size-cutoff to see full output.
Use -p '/T190/' to rerun this test only.
ShowDeriving: FAIL (5.12s)
295,296c295
< data SFoo1 :: Foo1 -> GHC.Types.Type
< where SMkFoo1 :: SFoo1 (MkFoo1 :: Foo1)
---
> data SFoo1 :: Foo1 -> Type where SMkFoo1 :: SFoo1 (MkFoo1 :: Foo1)
302c301
< data SFoo2 :: forall a. Foo2 a -> GHC.Types.Type
---
> data SFoo2 :: forall a. Foo2 a -> Type
331c330
< data SFoo3 :: Foo3 -> GHC.Types.Type
---
> data SFoo3 :: Foo3 -> Type
352c351
< -> GHC.Types.Type) t1) t2) t3)
---
> -> Type) t1) t2) t3)
369c368
< -> GHC.Types.Type) t1) t2) t3)
---
> -> Type) t1) t2) t3)
488c487
< -> GHC.Types.Type) t1) t2) t3)
---
> <truncated>
Use --accept or increase --size-cutoff to see full output.
Use -p '/Singletons.ShowDeriving/' to rerun this test only.
EmptyShowDeriving: FAIL (4.29s)
49c49
< data SFoo :: Foo -> GHC.Types.Type
---
> data SFoo :: Foo -> Type
63c63
< -> GHC.Types.Type) t1) t2) t3)
---
> -> Type) t1) t2) t3)
Use -p '/EmptyShowDeriving/' to rerun this test only.
StandaloneDeriving: FAIL (4.88s)
305c305
< data ST :: forall a b. T a b -> GHC.Types.Type
---
> data ST :: forall a b. T a b -> Type
316c316
< data SS :: S -> GHC.Types.Type
---
> data SS :: S -> Type
332c332
< -> GHC.Types.Type) t1) t2)
---
> -> Type) t1) t2)
351c351
< -> GHC.Types.Type) t1) t2)
---
> -> Type) t1) t2)
383c383
< -> GHC.Types.Type) t1) t2) t3)
---
> -> Type) t1) t2) t3)
415c415
< -> GHC.Types.Type) t1) t2)
--<truncated>
Use --accept or increase --size-cutoff to see full output.
Use -p '/StandaloneDeriving/' to rerun this test only.
T197: OK (4.15s)
T197b: FAIL (3.29s)
54c54
< data (%:*:) :: forall a b. (:*:) a b -> GHC.Types.Type
---
> data (%:*:) :: forall a b. (:*:) a b -> Type
65c65
< data SPair :: forall a b. Pair a b -> GHC.Types.Type
---
> data SPair :: forall a b. Pair a b -> Type
Use -p '/T197b/' to rerun this test only.
T200: FAIL (4.27s)
146c146
< data SErrorMessage :: ErrorMessage -> GHC.Types.Type
---
> data SErrorMessage :: ErrorMessage -> Type
Use -p '/T200/' to rerun this test only.
T204: OK (3.24s)
T206: OK (1.79s)
T209: FAIL (3.01s)
80c80
< data SHm :: Hm -> GHC.Types.Type where SHm :: SHm (Hm :: Hm)
---
> data SHm :: Hm -> Type where SHm :: SHm (Hm :: Hm)
Use -p '/T209/' to rerun this test only.
T216: OK (3.20s)
T226: OK (2.81s)
T229: OK (4.17s)
T249: OK (3.24s)
OverloadedStrings: OK (4.19s)
T271: FAIL (4.87s)
160,161c160,161
< (Sing n)
< -> SConstant (Constant n :: Constant (a :: Type) (b :: Type))
---
> (Sing n) ->
> SConstant (Constant n :: Constant (a :: Type) (b :: Type))
Use -p '/T271/' to rerun this test only.
T287: OK (3.38s)
TypeRepTYPE: OK (2.56s)
T296: OK (3.24s)
T297: OK (3.34s)
T312: OK (3.55s)
T313: OK (3.32s)
T316: OK (4.51s)
T322: OK (4.40s)
T326: FAIL (3.27s)
26,27c26,27
< type PC1 :: GHC.Types.Type -> Constraint
< class PC1 (a :: GHC.Types.Type) where
---
> type PC1 :: Type -> Constraint
> class PC1 (a :: Type) where
55,56c55,56
< type PC2 :: GHC.Types.Type -> Constraint
< class PC2 (a :: GHC.Types.Type) where
---
> type PC2 :: Type -> Constraint
> class PC2 (a :: Type) where
59c59
< class SC2 (a :: GHC.Types.Type) where
---
> class SC2 (a :: Type) where
63c63
< type SC2 :: GHC.Types.Type -> Constraint
---
> type SC2 :: Type -> Constraint
Use -p '/T326/' to rerun this test only.
NatSymbolReflexive: OK (1.99s)
T323: OK (1.84s)
T332: FAIL (4.26s)
59,60c59
< data SBar :: Bar -> GHC.Types.Type
< where SMkBar :: SBar (MkBar :: Bar)
---
> data SBar :: Bar -> Type where SMkBar :: SBar (MkBar :: Bar)
Use -p '/T332/' to rerun this test only.
T342: OK (3.38s)
FunctorLikeDeriving: FAIL (7.25s)
957,961c957,961
< (Sing n)
< -> (Sing n)
< -> (Sing n)
< -> (Sing n)
< -> ST (MkT1 n n n n :: T x a)
---
> (Sing n) ->
> (Sing n) ->
> (Sing n) ->
> (Sing n) ->
> ST (MkT1 n n n n :: T x a)
Use -p '/FunctorLikeDeriving/' to rerun this test only.
T353: OK (3.36s)
T358: OK (4.37s)
T367: OK (4.43s)
T371: OK (4.55s)
T376: OK (3.35s)
T378a: OK (4.47s)
T378b: SKIP
Use -p '/T378b/' to rerun this test only.
T401: OK (2.48s)
T401b: OK (2.71s)
T402: OK (3.50s)
T410: OK (3.39s)
T412: FAIL (3.66s)
163c163
< data SD1 :: forall a b. D1 a b -> GHC.Types.Type
---
> data SD1 :: forall a b. D1 a b -> Type
226,227c226,227
< type PC2 :: GHC.Types.Type -> GHC.Types.Type -> Constraint
< class PC2 (a :: GHC.Types.Type) (b :: GHC.Types.Type) where
---
> type PC2 :: Type -> Type -> Constraint
> class PC2 (a :: Type) (b :: Type) where
231c231
< class SC2 (a :: GHC.Types.Type) (b :: GHC.Types.Type) where
---
> class SC2 (a :: Type) (b :: Type) where
235c235
< type SC2 :: GHC.Types.Type -> GHC.Types.Type -> Constraint
---
> type SC2 :: Type -> Type -> Constraint
246,247c246,247
< type T2aSym0 :: (~>) GHC.Types.Type ((~>) GHC.Types.Type GHC.Types.Type)
< data T2aSym0 :: (~>) GHC.Types.Type ((~>) GHC.Types.Type GHC.Types.Type)
---
> type T2aSym0 :: (~>) Type ((~>) Type Type)
> data T2aSym0 :: (~>) Type ((~>) Type Type)
255,257c255,256
< type T2aSym1 :: GHC.Types.Type
< -> (~>) GHC.Types.Type GHC.Types.Type
< <truncated>
Use --accept or increase --size-cutoff to see full output.
Use -p '/T412/' to rerun this test only.
T414: OK (3.40s)
T443: OK (3.40s)
T445: SKIP
Use -p '/T445/' to rerun this test only.
T450: FAIL (4.55s)
50,51c50,51
< type SMessage :: PMessage -> GHC.Types.Type
< data SMessage :: PMessage -> GHC.Types.Type
---
> type SMessage :: PMessage -> Type
> data SMessage :: PMessage -> Type
119,121c119,121
< type PMkFunctionSym0 :: forall (a :: GHC.Types.Type)
< (b :: GHC.Types.Type). (~>) ((~>) a b) (PFunction (a :: GHC.Types.Type) (b :: GHC.Types.Type))
< data PMkFunctionSym0 :: (~>) ((~>) a b) (PFunction (a :: GHC.Types.Type) (b :: GHC.Types.Type))
---
> type PMkFunctionSym0 :: forall (a :: Type)
> (b :: Type). (~>) ((~>) a b) (PFunction (a :: Type) (b :: Type))
> data PMkFunctionSym0 :: (~>) ((~>) a b) (PFunction (a :: Type) (b :: Type))
129,132c129,131
< type PMkFunctionSym1 :: forall (a :: GHC.Types.Type)
< (b :: GHC.Types.Type). (~>) a b
< -> PFunction (a :: GHC.Types.Type) (b :: GHC.Type<truncated>
Use --accept or increase --size-cutoff to see full output.
Use -p '/T450/' to rerun this test only.
T453: OK (3.32s)
NegativeLiterals: OK (3.31s)
T470: OK (4.41s)
T480: OK (2.66s)
T487: OK (4.31s)
T489: OK (4.21s)
T492: OK (2.17s)
Natural: FAIL (4.24s)
63c63
< data SAge :: Age -> GHC.Types.Type
---
> data SAge :: Age -> Type
Use -p '/Natural/' to rerun this test only.
T511: OK (2.81s)
Promote
Constructors: OK (3.25s)
GenDefunSymbols: OK (3.08s)
Newtypes: SKIP
Use -p '/Newtypes/' to rerun this test only.
Pragmas: OK (3.26s)
Prelude: OK (3.88s)
T180: OK (3.76s)
T361: OK (2.96s)
Database client
Database: OK (5.43s)
Main: OK (1.97s)
InsertionSort
InsertionSortImp: OK (3.23s)
47 out of 122 tests failed (38.75s)
So maintaining two sets of golden files (one set for 9.2 and another for 9.4) wouldn't quite double the amount of files, but it would involve enough duplication to where I wouldn't be excited to maintain it.
Maybe the
tested-withbit in a .cabal file is sufficient to label this fact, though.
The tested-with stanza in a .cabal file is primarily used by CI tooling to determine which versions of GHC it's built against, not as a way to indicate the testing coverage.
In #531, I've decided to keep the status quo and require singletons-{th,base} to use GHC 9.4. I'm still not quite happy with this state of affairs, but it has become evident that we'll need to re-engineer the test suite in non-trivial ways to accomplish what I want, which is not something that should hold up a GHC 9.4–compatible release. I'll keep this issue open as a reminder to perform this engineering.