FrameworkBenchmarks

FrameworkBenchmarks copied to clipboard

Could a SqlServer DB be added?

Based on docker FROM microsoft/mssql-server-linux:latest

Just out of curiosity, what is licensing like for SQL Server on Linux?

The free Express edition would not be the right fit since Express is limited to 4 CPU cores. I suspect we'd want to use the Developer edition.

Same as Windows I believe; SQL Server 2017 brought parity between the editions in base db features/programmability (whereas previously whether you could use memory optimised, column store or data-compression was based on the edition - so Express was more a SQL Server Lite)

At SQL Server 2017, Express is now fully featured from a base db perspective; with editions upping core, memory, db size limits as well as adding "bigger database" features: scaling (failovers, clustering etc), analytics, auditing etc - so using Developer edition (which is Enterprise), should still be a valid representation of Express.

Going from the pricing page; Developer and Express are both free; with the caveat that Developer can only be used for build, test and demonstration - but not production.

So I think it would be correct to use Developer edition for the benchmark tests.

@benaadams We were just discussing this internally. We believe that Citrine qualifies as a "test" environment rather than a "production" environment for the purposes of licensing. Is it possible to get anyone at Microsoft who knows the SQL Server license model well to confirm this for us? We just don't want any bad surprises.

We will also need to contend with clause 6 from the license terms:

6. BENCHMARK TESTING. You must obtain Microsoft's prior written approval to disclose to a third party the results of any benchmark test of the software.

We will also need to contend with clause 6 from the license terms:

BENCHMARK TESTING. You must obtain Microsoft's prior written approval to disclose to a third party the results of any benchmark test of the software.

Oh wow, that's definitely an issue... @DamianEdwards any idea how to proceed?

Also earlier one on licencing:

We believe that Citrine qualifies as a "test" environment rather than a "production" environment for the purposes of licensing. Is it possible to get anyone at Microsoft who knows the SQL Server license model well to confirm this for us? We just don't want any bad surprises.

Does anyone watching this conversation know who we could contact on the SQL Server team to get the necessary waivers or permission to proceed?

Hi. I can handle getting the data reviewed and provide written permission post-review. I do this for all of our TPC benchmark results as well as our Data Warehouse Fast Track program.

@jamiere-msft We'd like to publish results continuously without anyone needing to review each individual set of results. Currently, we publish results automatically at https://tfb-status.techempower.com/ and manually at https://www.techempower.com/benchmarks/. The amount of SQL Server-related data in our results would increase over time, because any contributor to this project would be able to write a test that pairs SQL Server with some web framework of their choosing.

Is this model of benchmarking compatible with SQL Server's licensing?

Thanks Michael. I need to run this by a few people here to see if we could do a global permission for data coming from this workload.

@jamiere-msft did you have any luck with this question yet? Corporate decisions are often influenced by creating potential bad headlines as a test case ;-)

@mi-hol Yes and no. I reached out to @benaadams to see if he could share with us some preliminary data points to help our argument. Once we get a data point or two we can continue the internal discussions.

@jamiere-msft @benaadams I wonder if there might be news to share?

These benches would be handy for perf comparisons on work i've done. Would they have to use linux builds? the linux perf of SqlClient lags behind native

@benaadams any news about SqlServer?

@Wraith2 Currently, because of a bug in Docker4Win, the toolset is limited to Linux-only. This will, presumably, be corrected at some point.

I'd be very curious to see how it performs with MS SQL Server. Anything the community can do to help here?

no, waiting on Microsoft.

Still waiting on the MS side of things, an update would be useful given the performance work that i've put into Microsoft.Data.SqlClient

We're coming up to the 2 years anniversary of this thread and we're still waiting on movement from the Microsoft side on the licensing question. Given that the TE benchmarks are being used as optimization targets for much of aspnet core it'd be really good to get some benchmarks for the DB in there as well.

Another 6 months. Given how often techempower was mentioned a .netconf it's still a big omission that sql server isn't present in the benches. It gives the impression of a lack of confidence in the product.

Another round missed, https://www.techempower.com/benchmarks/#section=data-r20&hw=ph&test=fortune Will it make 3 years without anything happening? Looks likely.

3 Years old today.

4 Years old today! 🎉

Still waiting on Microsoft.

5 Years old today!

Still waiting on Microsoft. It is possible that the request has been lost internally. Does anyone know who to contact who is currently in a position to do something about this?