fix parsing of debian/copyright files

[tmuehlbacher-bnr]

What happened:

Too many Debian packages contain poorly parsed copyright information.
It's even contained in at least one test case:

syft/syft/pkg/cataloger/debian/parse_copyright_test.go

Line 35 in d71b747

// note: this should not capture #, Permission, This, see ... however it's not clear how to fix this (this is probably good enough)

What you expected to happen:

Only parse debian/copyright files according to the machine-readable format if they make the claim to be in that format.

https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ specifies that there is a mandatory field called Format: that contains a (http or https) link to the spec.

Only files that contain this field should be parsed as machine-readable. All other files should instead have their entire content put into the .text.content of a license object.

Steps to reproduce the issue:

It's part of a test case in the repo.

[kzantow]

Hey @tmuehlbacher-bnr, we talked about this on our weekly livestream; it sounds like we should definitely add support for the Format: field and look up the URL against our list of known license URLs as a first step, I think this should be straightforward for anyone to pick up. 👍

It would also be great if we could figure out a way to validate or parse short form license text into SPDX IDs, too, so we could determine if the text we found looks like a license we know about and maybe filter out some of the unknown stuff and fall back to using the license classifier when we haven't identified any known licenses. We have a related issue for the first part of that: #2210 -- if someone picks up this issue, falling back to the license classifier would be a nice addition in cases we didn't identify any known licenses.

[tmuehlbacher-bnr]

Thanks for looking into this, the live stream discussion was actually quite insightful to get a feel for where you're at with this.

IMO, the full, ideal impl I would imagine:

• The principal key/value format that Debian uses here and in many other places is deb822. I would suggest getting a parser (external or homegrown) for this, similar to like e.g. a JSON parser.
• Then the linked spec for machine-readable licenses is basically a schema in this deb822 serialization format.
• If you can impl the schema and parse a d/copyright file into this schema, which requires the Format: field in the first stanza, you can go on and parse the individual stanzas. E.g.:
  • Check if the License: field has an SPDX synopsis or can be fuzzy identified as a common license.
  • Potentially record the other relevant fields of the stanzas in properties.
• If you can't successfully parse as deb822, you could still try to fuzzily identify the content as some known license. However, the string may identify varied and multiple common and uncommon licenses in a way where it's not easy to accurately id completely correctly.

Some background, as I understand it:

Debian has a really large data set of copyright info, but unfortunately much of this data precedes SPDX and other specifications to be easily parsed. Even the machine-readable format was specified at some point before the big standards and many packages don't follow that yet.
Given the nature of these files specifying the copyright of countless entities, and the fact that all of this data is scattered among thousands of packaging repos on salsa.debian.org and elsewhere, it's quite a challenge. (e.g. I wouldn't feel confident to just come along and rewrite someone else's copyright file.)

I could be wrong, but I don't think there is much linting (through lintian) for following the machine-readable format. So presumably not all of the 1000s of Debian developers/maintainers will have gotten the format correct, even if they skimmed over the spec and attempted to follow it.