portmidi updates

[rbdannenberg]

Hi mixxxdj,

I'm the creator of PortMidi. I was a bit surprised to find how many portmidi versions there are on Github. I've finally gotten around to making a number of updates that are mostly minor but long overdue.

I also decided it would be wise to host a copy of PortMidi on Github.

I went through every copy of PortMidi I could find, looking at 4583 files and running diff 7961 times, but who's counting? :-)

I picked up a few fixes from your collective works, although with this many files and changes, I'm sure I missed something.

This appears to be mainly a clone of an earlier version from SourceForge (which is also updated). I wonder if you'd mind referring people to rbdannenberg/portmidi and/or SourceForge for the lastest version? Also, I see some reorganization and cmake changes here -- if these make the build system more general or more robust, I'd be happy to integrate them so we're not maintaining the same code twice. Let me know.

Thanks,
Roger

[Be-ing]

Hi, thanks for getting in touch and for writing this useful library! For context, the reasons I forked this was to get it building for Mixxx and Tenacity for macOS (and Linux) with vcpkg, the SourceForge project hadn't had activity in years and no release in a decade, and not wanting to bother with SVN or SourceForge. It also helped @jacardenasm package portmidi for FreeBSD instead of having to add a bunch of patches to the build system for the FreeBSD package. I would be glad to join our efforts. I don't especially care whose name the canonical repository is under. Maybe we could make a PortMidi GitHub organization instead of having it under your personal account? Looking at https://github.com/rbdannenberg/portmidi, I don't think that's a great way to start the project with Git. I used GitHub's import from SVN feature to import only portmidi and separate it from the other libraries in the SVN repository. That also avoids having the library under a trunk directory which doesn't really fit with how Git branches work.

I see you've made some recent changes to the code and quite a bit of them are for the build system. The build system in this repository was completely rewritten from scratch and is much simpler than the old build system. Notably, I did not bother getting any of the bindings working with the new build system; I am not aware of (currently maintained) applications which use them. Looking through the reverse dependencies in Arch, all of them are C or C++ applications except python-pygame. @dvzrv just switched the Arch package to this repository the other day. It looks like the python-pygame rebuild is still pending. I don't know if that uses the Python bindings from the original repository or has its own way of binding from Python. We may need to reimplement support for Python bindings in the build system, but I don't know if it's worth bothering for the Java, C#, and Lisp bindings.

Perhaps we could cherry pick the recent changes you've made to the C code into this repository. This may be a bit cumbersome with the reorganization I've done but I think it shouldn't be too much work. I just checked the Git history of this repository and I haven't made any changes to the C code besides this small fix for the new build system. The changes are all in the build system, reorganizing files, and removing no longer needed files.

In this repository, we have an open pull request (#6) for adding a backend for the sndio server used on some BSDs and available as an option in a few Linux distros. That needs some more work before it is mergeable though.

In terms of continuing to improve the library, the one major feature we are sorely lacking in Mixxx is hotplug support. Having the USB cable come unplugged during a performance and not being able to plug it right back in and have that just work is really bad! We have considered switching to a different cross platform MIDI library to get MIDI hotplug in Mixxx, but we could also work together on getting that working in PortMidi. Would you be interested in working together on that?

[Be-ing]

I see you've setup GitHub Pages to host the Doxygen generated documentation and put the generated files next to the code in the repository. A cleaner way to set that up is to have GitHub Pages use a different branch of the repository for the generated documentation so the autogenerated artifacts aren't mixed together in the Git history with code changes. Take a look at how @Holzhaus set that up for libkeyfinder, another dependency of Mixxx that we forked. The documentation generated by GitHub Actions is deployed to https://mixxxdj.github.io/libkeyfinder/

[Be-ing]

I see https://github.com/rbdannenberg/portmidi also contains portsmf, which I also forked to https://github.com/tenacityteam/portsmf (Mixxx does not use PortSMF). Like this repository, there are no functional changes; I only rewrote the build system, reorganized the files, and made a few minor fixes to the build (https://github.com/tenacityteam/portsmf/commit/951b636f7d0cba370d483a91f1897c71f3d98530 and https://github.com/tenacityteam/portsmf/commit/5b593d11f808be4e04d1cc54b14761ecd208d3b0). @Veratil also contributed some code cleanup addressing compile warnings.

Perhaps we could create a PortMedia organization on GitHub with a separate repository for each of these libraries? Keeping multiple libraries in one repository makes it cumbersome to make releases for each of them independently and makes the Git history more difficult to make sense of.

[Veratil]

Recently I've been looking at a large refactor/rewrite into more C++ code than C code wrapped in C++ classes. With this I've also tossed around the idea of just making it a separate identity, like a "portsmf2". Converting a lot into using the standard library will give a good amount of memory safety and reduced code amount over the current state.

[Be-ing]

@Veratil that is an interesting idea. Are you planning to keep it API compatible with PortSMF? If not, I'm not volunteering to change Tenacity to use the new API, but if you want to do so for LMMS, go for it. Regarding the name, I think PortSMF is a bit of an odd name. It implies that it is somehow a cross platform abstraction like PortMidi or PortAudio, but it is just plain portable C/C++ code that doesn't depend on OS-specific APIs. If you want to fork it to a new library, maybe "libsmf" could be its name?

[Veratil]

@Veratil that is an interesting idea. Are you planning to keep it API compatible with PortSMF? If not, I'm not volunteering to change Tenacity to use the new API, but if you want to do so for LMMS, go for it.

I was planning on it being straight API compatible, only replacing the building blocks of it with C++, but it doesn't have to be. There could always be a compatibility layer, too.

Regarding the name, I think PortSMF is a bit of an odd name. It implies that it is somehow a cross platform abstraction like PortMidi or PortAudio, but it is just plain portable C/C++ code that doesn't depend on OS-specific APIs. If you want to fork it to a new library, maybe "libsmf" could be its name?

libsmf is already in use as well as an extension of it. They are written in C though.

[Be-ing]

If the API won't change then I think it would be a good idea to keep the name of the library as PortSMF.

[rbdannenberg]

Are you getting me email? I seem to get some but not all GitHub notifications, so still trying to figure out an approach that works. Monitoring 15 social media channels, unfortunately, has never worked for me, so messages via GitHub are unreliable unless GitHub forwards them consistently. [email protected] works best. Thanks.

[Be-ing]

I have not received any email from you and I did check my spam folder. You already started the conversation here and I have invited more people into the discussion; I think it would be best for it to continue here in public.

[rbdannenberg]

Thanks - I see the problem is that GitHub forwards these messages to me (that's good) by email and says "Reply to this email directly...", but they are not delivering the mail, nor do they do not return it as undelivered. In the future, I'll log in here (github.com web service) and answer directly. I'll re-post the missing replies.

[rbdannenberg]

This is an (updated) reply to #8 (comment) (sorry it was dropped)

Thanks for all the interesting information. Do you have a name?

I have no objection to moving to a GitHub organization. UPDATE: PortMidi just granted admin rights so I can use it. I plan to move PortMidi and PortSMF there as separate repos, and put a copy of scorealign in rbdannenberg.

I also used GitHub to import the project from SourceForge, and didn't realize how it organized things until later. I was hoping for a clean one-to-one correspondence between SourceForge and GitHub (I don't understand why so many treat a proprietary service offered by Microsoft to be a de facto standard that everyone should use and follow, but I guess I've already decided presentation is everything, and if that's how people want their software distributed, so be it.) So reorganizing is fine. I'm going to keep a mirror on SourceForge, but if this becomes a group effort on GitHub, I'll automate propagating changes back to SourceForge and/or turn the repo there into multiple git repos.

Ultimately, I think bindings should be separate, dependent projects, and I added notes that current bindings are not updated or maintained. I didn't go so far yet as to actually remove all the code, especially Java, which actually removes the default system functionality (documentation for Pm_GetDefaultInput/OutputDevice() now explicitly says these are not recommended). Documentation also says let me know if lack of Java support is an issue -- I don't know if there are users or not. UPDATE: I guess the thing to do is strip out Java into a PortMidi/pmjava project too.

Agreed that Python bindings need work. I suspect someone has done them, and maybe I'll hear back since I wrote about 100 people and said if you have something useful, let me know and I'll recommend it from the portmidi README.md file.

I integrated preliminary Python bindings long ago, but did not get any feedback from any users/developers, so I don't want to repeat that experience. I'm very happy to point to the work of others or share an organization that has everything in one place -- that was the original goal of PortMedia, but it just didn't reach critical mass.

sndio looks interesting. Portmidi should support Jack too. With all the recent changes and after cherry picking from 100 repos (didn't find too much though), I think updating your sources would be more work than I want to get into. UPDATE: I created a live boot and grabbed the sndio code -- it seems pretty simple, so I hope to integrated it.

Hotplug support has a lot of issues. Most systems are very slow to respond to device queries, so if you are in the middle of performing, there's not much chance of switching devices without distrubing a lot of timing. I've seen that some people are actually polling RtMidi for devices to become aware of new devices, which seems very risky to me.

But I guess between performances, polling devices is reasonable. To do that in PortMidi, you'd have to know when, and you'd have to have persistent device identification rather than integer device ID's, which is a pretty big change. I've been working at the application level and under user control, using a preferences system to store user selections by device name. At least when you start the application, connections can be made automatically. I also usually just have one input and one output device and at the application level you just write MIDI and let some glue code send MIDI to the currently open PortMidi device, which can be reassigned with a GUI menu selection.

For PortMidi, it might make sense to add a layer of abstraction, named devices, and allow streams to be opened on named devices rather than device IDs. PortMidi could keep track of named devices, and when something like Pm_Refresh() is called, named device streams could be overwritten to reference a low-level device ID, so when you plug in a device with a matching name, the stream would autmatically start using it.

I think this would not be too hard and I'm willing to make it work. It would be great to get your input. I think it's important to get the right abstraction.

[rbdannenberg]

This is the reply to #8 (comment) on Doxygen:

Yes, I thought that was very strange for github to build this pages facility that suggests keeping generated documentation in the code repository. I never did that before, and only did it here because I'm trying to be more github-native, given the popularity of github.

Do you think people will use documentation if it's not in docs like github suggests? UPDATE: I see now that docs is just a special case for serving HTML that's part of a repo. I think removing the html and generating pages is a better idea.

One problem with Holzhaus' set-up is he has no links to documentation. I looked at his README and for a docs/ directory, but neither exists, so I never would have found it. UPDATE: But with easily added links, it's a better solution. Thanks.

[rbdannenberg]

Reply to #8 (comment) on PortMedia and separating repos:

Agreed - I wonder who has PortMedia? UPDATE: I did the best I could to communicate to GitHub but I don't know if they'll even forward my request. Meanwhile, the PortMidi organization is available.

[rbdannenberg]

Reply to #8 (comment) on tenacity development:

I think I might not be able to contribute to tenacityteam -- what a mess. How is tenacity doing? portsmf gets some love every once-in-awhile when someone finds a problem with something in Audacity. It's pretty complex, so there are few users, and not much use within Audacity, but I think it has much more functionality than many SMF libraries like the Mido or libsmf, e.g. there are cut/copy/delete/paste/stretch operations that respect tempo maps and time signature maps. It has been very useful to me and my students for research, so I will stick with it.

This makes me realize some non-trivial changes from other contributors to PortMidi or PortSMF could have some implications for Audacity. I guess I won't worry about it, and Muse will just have to sort out where new code originated if they want to use it.

[Be-ing]

I think updating your sources would be more work than I want to get into.

I would be happy to copy your updated C code into this repository if we agree that's a good way forward. Then you could start the new repository under the PortMidi or PortMedia GitHub organization by simply pushing this repository to a new remote.

[Be-ing]

I have copied your updates into this repository in commit 7c82f88 which I merged into the main branch of this repository.

Yes, GitHub is proprietary and an awful company, but GitHub Actions is very helpful for maintaining cross platform code. I didn't have to manually build the code on three different operating systems to test if copying your recent changes into this repository broke the build; GitHub Actions did that automatically within a minute of pushing the update.

[Be-ing]

I would be glad to work towards hotplug in PortMidi. I don't want to get too in depth into the technical details for that here; let's setup the new repository and continue that discussion on a GitHub issue.

[Be-ing]

It looks like there already is a PortMidi organization on GitHub: https://github.com/portmidi

[rbdannenberg]

It will take a bit to sort through all these different options and do some testing -- compiling seems like the quick and easy part compared to testing. I became PortMidi organization admin today (!). I also grabbed PortMusic (organization name), but I think PortMidi is better since it has many more users.

[illume]

Glad to see movement to a github organization! (hello from pygame)

[Be-ing]

@rbdannenberg, how do we proceed here? You can move this repository to another on GitHub simply by cloning this, adding the new URL as a remote, and pushing to the new remote.

[rbdannenberg]

The portmidi organization has portmidi with some of the changes we talked about. Discussion with @marcdinkum revealed a design problem with virtual device creation: creation should not open, and we already have open functions. I'm working on it. I don't want to replace the repo because then I have to go through all the differences to assess every change.

[Be-ing]

The changes are only in the build system and reorganizing the files. This is plain to see from the Git history.

[Be-ing]

Since my offer to start from this repo was rejected, I have reimplemented the reorganization and new build system in the other repo: rbdannenberg/portmidi#2