PartKeepr
PartKeepr copied to clipboard
partkeepr
About the future
Hello,
First I would like to thank Felicia. She did a wonderful job for 10 years building this tool.
[Partkeepr as of 1.4.0]
As we can all see, this project is dead and stuck in Symphony 2.
The docker image is fully working but as some users pointed out, this leads to security issues and any feature request can be easily implemented. Octopart support is now also dead as Altium's API is very expensive.
Some users tried to move to Symphony 3 (which is outdated too) without any relevant success as a lot of things must be rewritten. This rewritten things would of course need to be rewritten again while updating to a newer version of Symphony.
I guess this project is stuck because of the legacy technology : as the 2 years not merged PR shows, writing old PHP / Symphony code is a rare and niche talent that few of us have.
[Why not re-writing ?]
Back in time when this project started, server side JavaScript didn't exist. Now building a web based app with database management schema like Partkeepr is a lot easier with Node.js.
If some of you are interested (and have some knowledge in Javascript), maybe it's time to start a full rewriting with a newer, easier and maintained technology.
Echoing the thanks to Felicia!
Having built some homebrew solutions for this in the past, wanted to offer help in reimplementation efforts. Anyone else game for this?
I've made some research and asked few friends (pro web dev) about the best technology for a reimplementation. As for me a solution based on nuxt.js / node.js / mysql could work : nuxt (vue.js metaframework) make things easy for the frontend (and there is a lot of documentation too) and node can make the link between the database and the frontend.
We would need to copy the database tree.
I'll try to make a quick graphic demonstrator if I have some band pass.
Any folk interested in a javascript reimplementation, manifest yourself here.
I have quite a lot of experience with Go, which I believe would be an even better choice for the backend. The backend shouldn't take too much time, I guess the UI would take a lot of work to create if it is being rewritten.
I'm a relatively new programmer.... okay so I've really been on/off for like 10 years and never really progressed much but I'm starting to really gain interest in React, NoSQL, docker, and Python (including Flask... never did much with Django). I haven't dug into node much but I'm willing to learn and would be interested in helping get this project going again. I don't know how much help I might be, I may just get in the way, but if this gains any traction, please let me know. I'd love to help. And echoing above, I never had a need for PartKeepr but knew it existed and finally had a need and knew right where to go.... it's a well known project, great job to all the hard work the dev team put in here.
Hello to you all, guys. I just wanted to keep you posted on the most recent status and thoughts we were having.
First, it is nice to hear that there are at least some folks out there that are willing to help with the problem. From your posts, I see at least experience in NodeJS, Go, and Python. That is good! It is also good that there are still users interested in the project. That means it is not dead. Currently, the development has stalled.
There were a few internal discussions going on how to proceed. Let me give you an overview.
Update by external resources
We had the idea to use a crowdfunding project to collect some money and let a pro do the Symfon 2 -> Symfony 5 migration. We had contacted a pro developer and have already a first quote that might be manageable if a few dozen people participated. We are also trying to get a second offer at least for comparison.
Once we have them, we wanted to reach out to you as a community but also to other communities in the hacker scene hoping to raise the required funds. This would mean a bigger PR campaign that would be needed.
The problem here is that we do not have a clear communication strategy for the community in general: The IRC channel is mostly empty (only a few people are there hanging around). In the google list, there are around 150 members. I suspect there are many more users out there that might benefit from installing and be willing to help.
If you have any good ideas on how to mobilize as many users as possible, feel free to speak up.
Discussions about rewriting
We also thought about what could be done to
- bring the project into a running state again
- ensure future management is easy
- allow future enhancements and extensions
- while keeping the huge work made by Felicia as best as possible.
One thing to identify is that the frontend (what you see in the browser) and the backend (the connection between HTTP and the database) are in fact separate projects/issues.
Felicia has defined a common interface between those two that is working great at the moment. This does not mean we have to stick completely with it. Adoptions/corrections/additions might be needed once development progresses. But I'd rather avoid building a complete architecture from scratch. That means we should stick for now with the API designed by Felicita.
Frontend update
As mentioned earlier, one could think of updating the frontend with some other framework. The currently used Ext.js might or might not be "the best" solution. There are for sure alternatives like VueJS/Nuxt.JS that might be more modern. Also, one could think of updating the building environment to e.g. webpack (or vite in case of Vue).
When looking at the most pressing problems, the Frontend seems mostly unconcerned as far as I see it. As long as we keep it consistent with the defined API, any updates can be done and even a parallel (call it modern) frontend might be possible.
Backend update
Here comes the big issue at the moment. We all agree that the current Symfony 2 is too outdated to be maintained anymore. Nothing is going around that. We know it.
The update of three Symfony versions is hard for a newcomer. This is especially true, as some backward-incompatible changes were introduced and the old documentation is only partially available nowadays anymore. So, you need, simply speaking, an expert in the field that knows the bells and whistles that can be used there.
If the update was not possible (lacking manpower and no intention to use external resources, see above), we have to think of replacing the backend. I am not telling you to rewrite from scratch. I am telling that the shiny new backend would have two interfaces to comply with: The database (including its current structure) and the API towards the frontend. If we managed to get this running, we could drag&drop replace the current backend with something else. If we do that we can migrate easily and even do a parallel mode similar to the frontend.
This "something else" might be written in different languages: NodeJS, Go, Python, Java, Brainfuck, or even PHP with Symfony 5 :smile:. Which language to use is (at least partly) not a trivial decision. Some languages are better suited, some are worse. The most pressing thing here is: We want to avoid focusing on one person that might drop out for whatever reason and let the same effect happen again we are facing at the moment. For example, if we have only one Java developer, that would be a rather bad choice.
I had recently a video call from a company from Germany that was intending to hack a bit on the database of Partkeepr. They wanted to have some information and were not aware that it was possible to replace the backend as I just presented. As they have experience with Python Django, they are intending to build their own backend for their needs. If we could use a cooperation partner as a bigger player like a company (in contrast to free-time contributors), I think that would be beneficial as the timing requirements are completely different.
Before you get too excited that you might get your favorite language (which you invited during your studies of computer science or learned somewhere else): If we decide to change the language, a few considerations must be passed! Here is an incomplete list of what comes directly to my mind:
- Changing language will cause lost knowledge by the old maintainers. For example, I am not experienced in NodeJS. So there need to be other people serving as quality gates for future PRs. So, you will have to commit in the long term.
- Updating only part of the core of Partkeepr is no solution. We need the complete backend. So, you will have to build all parts of the current structure in the new language. This means including all features.
- Such a big refactory might only be possible with automated tests (which we are lacking at the moment). This is one of the reasons, why the Symfony update is such a critical thing. No one wanted to take the risk of the update in the past (where it was just the current state of code). Now we have to fix that past error.
- Maintainability by the users: Our current users are mostly hobby users and a few institutions as far as I know. Some are running Partkeepr off their Raspberry Pis. We have to be careful not to require too much technical knowledge for the installation process and keep the runtime requirements at bay. (Does docker run on a pi? How complicated is it to compile Go on different platforms? What requirements are posed by NodeJS on Windows?) We do not want to lose many users in the process, so we have to keep things "simple".
You see, this is only possible with probably a group of developers that are committing themselves to the project in the long term. There needs enough experience to overcome some shortages. At the beginning of the rewrite huge amount of code need to be written and tested. So, to those of you, who are crying out for a complete rewrite: Are you willing to take this responsibility? Are you committing personally your time to the project? Are you willing to offer your free time for the next months or even years to the project? Can you promise there are no upcoming changes in your life? If you say full-hearted "yes" to all these questions, you are probably ready to take the responsibility of a public open-source project, possibly the rewrite of PartKeepr. If you are not sure, you might be better off in a community that is working together and cooperating towards a common goal. You are very welcome with suggestions, PRs, comments, testing, and anything else you can help with. I do not want to demoralize you all. I just want to tell you what it takes to make such a project. Felicita will be able to tell you a story about it. Even though she is strong, in the end, she was no longer able to support the project for different reasons. I just want to advise you in good faith and without offense to not waste your time when you are not sure you can make it.
How do we go on from here?
I highly recommend that we do not start to split up our community even further and have different versions of the backend running with different frameworks/languages/... This will not help the project at all. We are all looking for a solution fitting best to ourselves, but I think it absolutely essential that we try to work towards a common solution.
On the way there we need some objective way to decide what is good or not. Statements like "xxx is best" or "xxx is best because everyone uses it" do us no good in this discussion. We need hard facts. I have my personal preferences as well but for the sake of the project I am willing to participate in an open discussion about what options do we have.
Also if such a discussion starts, I would like to take the complete community into account. Maybe there are some hidden talents that could be used?
Call for action
- Do you know of any non-official channels to reach as many users of Partkeepr as possible? Please send your ideas.
- Do you know of any developers willing to help on the project? Send them here!
- Fill in this form if you are willing to help. It is a survey regarding possible experiences.
Thanks a lot for the response! As said above, I think for the backend Go would be the best choice:
- Can be cross-compiled for almost any platform*
- Does not need any dependencies*, results in a single binary
- Is faster than PHP, JS, Java or Python
- Is very fast to write and robust due to strong typing
- Has a lot of required components like http server, html templating and testing built-in
- Very fast compilation
- Easy to learn
The biggest disadvantage is however that there are probably a lot less people to program in Go - as you said you need to reach more than a couple of people in order to rewrite and maintain the project. Also Go binaries tend to be a little bit larger, I assume for this project they would be around 30-50MB uncompressed, which is the drawback of not requiring any dependencies. Also it would not possible to host Partkeepr on hosts anymore that only support PHP/mysql and are normally a little bit cheaper/userfriendly than a VPS instance.
*if no C code is included, which might be required to support sqlite3
Hey All!
Endless apologies for the lack of organization and communication the past 2 years.
I had sort of "inherited" the project with admin access to this repository from @Drachenkaetzchen Even though I've always, at this point almost a decade - sjeesh - been just a "power user" running several "custom" setups. I've never been at all a developer of the project and code-base.
Together with @christianlupus (thnx for the long post Chris! I fully agree to all points you made) and some other community "power user" members we initially got some work done to organize the project with some minimal improvements, mostly on the Issue tracking and CI side of things. Getting some bugfixes merged, etc.
Of course then the pandemic hit and everybody their lives where turned upside down. (I personally got pulled into a completely different project doing python/puredata/c/c++) However none of us are devs of the project and thus progress on pretty much all the things related to Partkeepr got stalled.
So here we are.
What can we do to keep this project alive?
Do we keep brute-forcing dependency upgrades as in https://github.com/partkeepr/PartKeepr/projects/3 and https://github.com/partkeepr/PartKeepr/pull/1214 ? Or first try to merge as much of https://github.com/partkeepr/PartKeepr/pulls as we can?
We can also try to find a way to replace the backend of the project on top of the current DB schema, but then the question is also if we should focus on "emulating" the current API?
If we're going to replace the stack, then I would actually prefer to move towards an OpenAPI Spec so that we can at least build in interoperability from the start. And alternate frontends could be created then too. However the current UI and API is so feature-complete. I don't know how easily we could instead have the current FE/ext.js converted to another API interface. Some better, interoperable, separation between the two would certainly be a win I think.
Personally I'm coming from mostly a Python (and Lua/OpenResty/Nginx) background and can't help support any PHP, Node or Go development. However I can of course help with testing, time permitting. For me I would then lean towards a framework like FastAPI. It implements the OpenAPI stack, simple, fast, with extensive typing and self-documenting. Anyone that wants to help with that hit me up.
Any suggestions on how to organize this are welcome. I'm certainly willing to add some more people access to the repository, to organize issues, and we can add some more projects to track things: https://github.com/partkeepr/PartKeepr/projects
Also come on IRC just to chat with other users if you want. Should be possible to add a matrix bridge as well. We're with a ton of users and there is nothing out there like PK so I really hope we can keep the project alive.
[Edit: I've pinned the topic. Maybe we should move this towards a "discussion" format. What do you think?]
@Forceu I really appreciate Go projects like Gitea btw! Single distributable binary that just contains pretty much everything you need.
Hmm, since the spec is based on Hydra, maybe converting the API to https://github.com/HTTP-APIs/hydrus (Python3/Flask/Sqlalchemy based) might be the easiest, for the Python heads?
The idea of having PartKeepr with semantic web capabilities is interesting. Imagine being able to link different "departments" or "communities" with access to certain parts of the api. RDF based specs are sometimes a bit hard to read, but it can be damn useful in distributed resources on the internet.
Please note that the frontend relies on entity information generated by the backend. The API and DB Schema are also generated by the entity information. The idea was to have one specific place where to define how an object looks like and how it's relations are - which are the entities. Everything else is derived from that.
I strongly advice against replacing the backend. If you want to do that, you basically need to re-write everything. PartKeepr was designed to allow filtering and sorting for every field and every relation, and there are no comparable projects that I know of.
I also disagree that PartKeepr should be re-written in a "faster" language. The demo system on https://demo.partkeepr.org runs on a low-end VM with little memory, and it's still fast despite the amount of data. If you need something faster it'll come at the cost of dropping the generic query system which can filter and sort everything, and you need to start to use indexes. A "faster" programming language won't help. A PartKeepr installation isn't used by thousands of users simultaneously.
The project took me 6 years to get it to the state where I considered it somewhat feature-complete for my requirements, and I did it with a clear vision what it should do and how it should do it. Replacing the backend or the frontend will easily get you over these 6 years I spent on that.
What this project needs is people willing to dig into the existing code, understand what it does, and go on from there.
@Drachenkaetzchen Thanks a lot for your insight and of course a big thanks for this great project! It has helped me a lot to organise all my electronics. You are right, the language does not have to be fast, and the search is indeed very convenient and quick. In the end I think you are right that a rewrite will take a lot of effort and it would be more efficient to update deprecated parts of the code.
Out of interest, why not use indexes?
Out of interest, why not use indexes?
You can't have an index for every field combination a user potentially wants to search.
This might be becoming off-topic now, but couldn't you use a word-list? Basically for every unique word in any category there is an entry that links to the item IDs - that way you would have a very fast search and can have custom categories as well.
This might be becoming off-topic now, but couldn't you use a word-list? Basically for every unique word in any category there is an entry that links to the item IDs - that way you would have a very fast search and can have a custom categories as well.
I never had the need for that. PartKeepr's search speed was always fast enough. Let me quote myself:
If you need something faster it'll come at the cost of dropping the generic query system which can filter and sort everything, and you need to start to use indexes.
How you would do it in this hypothetical scenario is then up to you.
The last time I implemented something like this homebrew I actually used google sheets as a backend, but they removed part of their scripts API for that.
Ever heard of Joplin? It's written on electron in js/ts and syncs to whatever db you set. I really like the idea of making this db agnostic to a degree and allowing users to host their data however they want as long as there's a plugin / parser between our API and whatever storage.
On Fri, Apr 15, 2022 at 1:49 PM Felicia Hummel @.***> wrote:
This might be becoming off-topic now, but couldn't you use a word-list? Basically for every unique word in any category there is an entry that links to the item IDs - that way you would have a very fast search and can have a custom categories as well.
I never had the need for that. PartKeepr's search speed was always fast enough. Let me quote myself:
If you need something faster it'll come at the cost of dropping the generic query system which can filter and sort everything, and you need to start to use indexes.
How you would do it in this hypothetical scenario is then up to you.
— Reply to this email directly, view it on GitHub https://github.com/partkeepr/PartKeepr/issues/1239#issuecomment-1100387601, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADMMNXAC6GUYUTNGEOZVPMTVFHI3XANCNFSM5QVS5YYA . You are receiving this because you commented.Message ID: @.***>
This might be becoming off-topic now, but couldn't you use a word-list? Basically for every unique word in any category there is an entry that links to the item IDs - that way you would have a very fast search and can have custom categories as well.
And in the case of "give me all resistors with a resistance > 400 and resistance < 500 in the package 0603" a word list won't do.
The last time I implemented something like this homebrew I actually used google sheets as a backend, but they removed part of their scripts API for that.
Ever heard of Joplin? It's written on electron in js/ts and syncs to whatever db you set. I really like the idea of making this db agnostic to a degree and allowing users to host their data however they want as long as there's a plugin / parser between our API and whatever storage.
Partkeepr should have its own independent database, I don't think it would make sense to use a 3rd party database.
Modern PHP is actually quite fast. Getting current PK codebase to that modern state though .. needs some many months of dedicated work and I'd rather see a team than a single person take on such a role.
However, even if we outsource/crowdfund such an endeavor we would still end up with a highly complex code-base that nobody can, or wants to maintain. What we want is a sustainable long-term solution for the project. One that meets business compliant practices (latest security patches and proper SSO/ACL).
I'm willing to donate 3-figures to have someone, or a team, work on something like this. But how do we make sure that there is a core team that will do all the required maintenance and upkeep?
+1
ping to follow progress. not mainly a web dev, can help with testing / small bug fixes though!
Even though I am just a hobbyist user, I believe PartKeepr has a future and would be willing to donate ~1000€ for a continued/restarted development effort (or smaller regular amounts). I sincerely believe PartKeepr deserves it.
But how do we make sure that there is a core team that will do all the required maintenance and upkeep?
That is also my biggest concern. Though hopefully by moving to a current technology stack, there is a larger pool of people with that kind of experience, reducing the complexity of finding a developer.
I agree that funding to support at least the latest version of symphony is mandatory and could represent the least expensive (money, time) option. We first need to find a web develloper that will accept the deal and gives us an estimate in hours.
We will then need to raise the funds.
To everyone here interested in funding, tell us how much you can give the project for a full symphony 6 upgrade.
I'll be giving ~50€.
I am also a hobbyist and currently looking for a part database. I looked on many free and even commercial products, but partkeepr (even in its current state) is exactly what I am looking for and would be my choice #1.
Most probably I will start using it for my home use, hopefully there are not too many bugs I have to deal with (I can correct small ones as I have some experience with php / python / c, but I am far away from being a real developer as I took the admin road in IT business).
However, I would like to see this project being renovated / resurrected. I would be willing to donate also ~50 EUR, which isn't too much when you have a look on commercial solutions with monthly subscription fees.
Thanks for the great work so far.
I've no experience which platform gets the most traffic from web devs - maybe someone else can chime in and set up a bounty?
I'm in with 250EUR myself.
I wouldn't mind to donate money either to upgrade this project since most of the alternatives I found don't suite my needs...
I would donate 100 Euros as well. It would be a good idea to setup a proper bounty. I found bountysource.com which seems good, but they do have a 10% fee for cashing out.
@Forceu do you think, that percentage is worth the extra attention we cold gain? (I'm not sure, but inclined to rather loose 10% than nothing happening)
@christianlupus do you mind to weigh in on this one?
The 10% is for the amount when the bounty gets awarded. Personally I thought 10% is quite a lot, but I also do not know if there are better alternatives.
Hello and sorry for the delayed answer. I was on a business trip recently and did not find a way to get online.
I am not sure if a bounty on the issue will get the required attraction of a pro (which I suspect we need). Let us go the other way around and collect the money (as a first guess) to acquire a programmer capable of solving the issue. Then, we could use a crowdfunding portal (like Kickstarter etc) to collect a predefined amount of money and have a clear responsible person for the task at hand.
Which portal is a good choice, I have no clue yet. I would have to do some research on my own. I think I once had a list but I have to refind it.
In general, I would be willing to insert the project with something like 150 to 250 €.
+1 to follow this topic. Not really a programmer but perhaps I can be of use in other areas. Pitching in with EUR 100,-- or recurring monthly with a slightly lower amount
[edit] On second thought, this tool could potentially be very, very usefull for hacker- and makerspaces. They often have large and diverse electronic stocks and these communities have the kind of contributers that could be beneficial to this project. With a slight adaption to their needs, it could also make this project sustainable in the long run...
