14:30:19 #startmeeting Pulp Triage 2019-12-10 signing in pulp 3 14:30:20 #info dkliban has joined triage 14:30:20 !start "signing in pulp 3" 14:30:20 Meeting started Tue Dec 10 14:30:19 2019 UTC. The chair is dkliban. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:30:20 Useful Commands: #action #agreed #help #info #idea #link #topic. 14:30:20 The meeting name has been set to 'pulp_triage_2019-12-10_signing_in_pulp_3' 14:30:20 dkliban: dkliban has joined triage 14:30:42 https://docs.google.com/document/d/1HSrRduFYuidhbc8ISjUnkb3IXk6FnJ2iXEqa4vDC-pA/edit# 14:30:56 ^ is the document where we have been keeping track of the discussion so far 14:31:37 #info x9c4 has joined triage 14:31:37 !here 14:31:38 x9c4: x9c4 has joined triage 14:31:47 #info dawalker has joined triage 14:31:47 !here 14:31:47 dawalker: dawalker has joined triage 14:32:05 #info misa has joined triage 14:32:05 !here 14:32:05 misa: misa has joined triage 14:32:33 #info dalley has joined triage 14:32:33 !here 14:32:34 dalley: dalley has joined triage 14:32:40 x9c4: want to re-ask your question here? 14:32:42 dkliban: s/key signature/signing key/ 14:32:54 Sure. 14:33:04 What do you mean by 'key signature'? Should this be 'key id'? 14:33:16 x9c4: haha - that's what I was correcting above 14:33:25 yeah ... that's what i mean 14:33:29 i'll update the doc now 14:33:40 misa++ 14:33:40 x9c4: misa's karma is now 23 14:33:58 #info bmbouter has joined triage 14:33:58 !here 14:33:58 bmbouter: bmbouter has joined triage 14:33:59 misa: I was asking in the wrong channel appareantly 14:34:13 #info ggainey has joined triage 14:34:13 !here 14:34:13 ggainey: ggainey has joined triage 14:34:29 so today i want to dig into the details of what the Pulp administrator needs to provide to the django-admin command in order to configure a generic signing service 14:34:51 i want to refer to pulp_deb again ... https://github.com/pulp/pulp_deb/tree/1.10-release#signing-support 14:35:29 sounds good 14:35:37 dkliban: I propose, like in the deb case, for the actual signing infrastructure to be a shell script that wraps whatever implementation is needed 14:35:55 and the shell script has to implement a certain interface 14:36:21 #info daviddavis has joined triage 14:36:21 !here 14:36:21 daviddavis: daviddavis has joined triage 14:36:30 #info ttereshc has joined triage 14:36:30 !here 14:36:30 ttereshc: ttereshc has joined triage 14:36:34 maybe "sigserv --sign file", "sigserv --list-keys" etc 14:36:44 where sigserv is a shell script on the filesystem 14:37:48 a signing service is an entry in the DB, with the path to the shell script 14:37:58 for clustered installs it'll have to be somewhere in the shared filesystem 14:38:17 that's in /var/lib/pulp 14:38:23 bmbouter: sure 14:38:41 unless the data is not there at all (like in the case of an HSM) 14:39:04 for us right now, it's an invocation of relic, which is the central signing server 14:39:22 but I think with a path on the filesystem, you can cover both cases just fine 14:39:41 the HSM would only be avaiable to one hardware unit in the cluster? 14:40:09 this script needs to be able to accept a path to a file in /var/lib/pulp/artifact/ and it needs to produce a signed artifact in the workers working directory 14:40:11 x9c4: in our current implementation it's a standalone service 14:40:47 x9c4: i would expect a pulp worker to send a request to a separate box that has HSM 14:40:51 So no need to specify 'signing' as a capability of certain pulp_workers. 14:40:58 x9c4: that's right 14:41:33 x9c4: relic is a client/server architecture. All workers have the relic client installed. The clients talk to a single (cluster of) servers 14:42:21 misa: +1 14:42:28 misa: in pulp_deb, there are a few environment variables that are made available to this script 14:42:32 x9c4: if one chooses to use straight gpg, then the gpg key store has to be either on the shared filesystem, or, of completely static, it can be anywhere (and configured with ansible or something) 14:42:50 dkliban: yes, those are for the plugin writer to fill out 14:43:00 ... or synced to all worker nodes. 14:43:11 x9c4: yep, that's what I mean 14:43:29 dkliban: in our case, we infer the key id based on the repository name 14:43:40 dkliban: to accommodate a dev/test/prod scenario 14:43:53 dkliban: since in pulp 2 there's only one singing service per plugin 14:45:24 bmbouter: do you think we should pass the repository name to the script via command line arguments or environment variables? 14:45:26 In pulp3 the key_id could be a property of either the repository or the publication. 14:45:39 dkliban: this is a good question. With repo admins being able to configure the signing key per repository, maybe the env vars are not that interesting anymore 14:45:57 dkliban: command line args are painful 14:45:58 yeah I think explicit args are more direct 14:46:17 misa: I'm confused which you're in favor for 14:46:18 dkliban: because processing command line options in bash is painful 14:46:46 bmbouter: I was proposing env vars 14:47:10 it's the hype du jour with 12 factor apps and docker and whatnot :-) 14:47:18 lol 14:47:39 so the command line args are just for the artifact(s) to be signed 14:48:17 any extra info that might be useful to the signing service to determine which key to use would be an environment variable 14:48:57 you can pass repo name as one of the obvious ones, but a plugin writer may add extra ones, and the signing service can use them 14:49:22 Which brings up a good question: detached vs. inline signatures, ascii armor or not, etc 14:49:40 so i think that should be an option .... 14:49:53 pulp_deb needs detached _and_ inline. But i'm ok with two calls for it. 14:49:54 env var :-) 14:50:21 the plugin writer should have 2 interfaces ... one for creating a new signed artifact. another one for creating an artifact that contains jsut the signature 14:50:58 dkliban: not sure I see the distinction 14:51:07 dkliban: you mean sign-on-create vs. sign after? 14:52:28 misa: i was thinking that there may be a need for keeping signatures separate from the artifact 14:52:53 Should the script really know more than "file to sign" "key id" and "type of signature"? 14:52:53 dkliban: are these artifacts present in the DB? If so, mutating them will be a pain 14:53:01 it's a use case listed under the Plugin writer use cases at the top of the document 14:53:24 misa: we don't mutate artifacts. we only create new ones 14:53:51 so there will be an un-signed artifact and a signed version of the artifact. they will be 2 distict artifacts 14:54:12 And it's easy to have a contentunit containing both the original artifact and its (detached)signature artifact. 14:54:16 x9c4: maybe also destination in case it's a detached sig 14:54:31 right. 14:54:32 x9c4: I believe some expect .asc and others .gpg or something 14:55:41 x9c4: what do you mean by destination? 14:55:46 oops ... misa 14:55:55 so the plugin writer should be able to tell the signing service what kind of an extension to put on the signature 14:56:17 Better specify the whole filename. 14:56:29 x9c4: agreed 14:56:31 yeah 14:56:43 dkliban: trying to find the example 14:56:44 OTOH it is saved as an artifact, and artifacts don't have a filename. 14:57:18 x9c4: yeah ... i think it's up to the plugin writer what the file should be called 14:57:22 It is up to the plugin again to serve the artifact with a relative path. 14:58:13 so as far as the plugin writer is concerned, the result of asking a signing service to sign an artifact will be a new artifact 14:58:31 and then the plugin needs to do something with that artifact 14:58:37 agreed. 14:58:52 dkliban: do you want to see our existing signing script for debian in pulp 2? 14:58:59 misa: sure 14:59:14 https://paste.centos.org/view/89acbd18 15:00:18 dkliban: I played with signing repomd.xml too, I believe that one expects a .asc - but we didn't push it to production, and I don't have an example of that 15:00:59 that's alright ... this already gives a good example 15:01:11 And for the signing script it is much the same it is provided a file an it is supposed to generate a new file. 15:01:27 the shell function contains() is basically string matching snapshot/milestone within the repo name 15:02:38 Again, with per-repo configs, some of that complication may be avoided 15:03:16 What I hear is that the interface to the script should be something like {script} {file-in} {file-out} 15:03:36 and maybe a 'detatched' boolean? 15:03:57 with additional env vars like GPG_REPOSITORY_NAME, GPG_KEY_ID, GPG_SIG_DETACHED 15:04:02 which could be part of the otherwise cleaned env. 15:04:13 oh ok 15:05:04 dkliban: in pulp 2 we were working with the limitation of one signing service (per plugin) 15:06:15 dkliban: since one can define as many signing services in pulp 3, things may be simpler - you may have a deb signing service that knows it creates a detached sig, and an rpm metadata signing service that produces inline (for example) 15:06:21 And you could only specify the key id in the plugin_config iirc. 15:06:47 x9c4: yes, or infer it dynamically like in that example 15:07:03 x9c4: but in pulp 3 we plan to let repo admins specify the key if they so choose 15:07:53 Yes, i like that. 15:09:02 this sounds good to me also 15:09:07 dkliban: so I would suggest a simple command line interface that is uniform across all plugin types, and additional env vars that plugin writers can use to pass information to the signing service 15:09:18 But the plugin could infer the key id from whatever and it has the freedom to sign whatever artifacts it desires. 15:09:46 x9c4: (and not just sign :-) 15:10:35 x9c4: it turns out that the difference between a yum repo and a *proper* zypper repo is an additional XML block in repomd.xml 15:10:55 wow 15:10:59 x9c4: so, we toyed with the idea of adding the XML block in repomd.xml using the signing infrastructure 15:11:13 even though it has nothing to do with signing 15:11:38 signing is a post-processing activity 15:11:47 just like splicing a block of XML is :-) 15:11:48 So it is a generic artifact processing infrastructure. 15:11:51 but we digress 15:11:59 exactly :-) 15:13:34 OK, but signing might have another caveat. How do we decide, which user/respository is allowed to use which keys? That question does not come up for splicing xml... 15:14:38 we need RBAC before we can worry about that 15:15:01 well, we can worry now, but it's hard to come up with a solution 15:15:57 What i mean is: If the plugin can provide arbitrary env vars, it might be impossible to impose those restrictions later. 15:16:04 What I'm trying to say is, if we don't impose too many restrictions on the signing service, it may be useful as a general purpose artifact processing infrastructure (good term x8c4) 15:16:23 x8c4 -> x9c4 15:16:26 :) 15:16:27 (: 15:17:23 x9c4: the signing service may still determine the config is inconsistent/bogus 15:17:48 for instance, the repo admin may ask for a signature with a key that either doesn't exist, or no longer exists 15:17:51 Ok. 15:18:06 in which case something has to alert that the publication cannot continue 15:18:46 The RBAC would be that only certain users would be allowed to use certain keys 15:19:06 which means... that signing keys need to be bona fide entities in the DB 15:19:52 (for the same reason signing services need to be bona fide entities) 15:20:14 yeah 15:20:30 that's beyond our initial imlementation for 3.1 15:20:32 so to summarize what we just discussed ... 15:20:41 definitely out of scope 15:21:10 key RBAC can be solved easily with signing service RBAC and more services, one per key 15:21:25 s/solved/worked around/ 15:21:40 Pulp admin is going to create a Signing Service by providing a path to a script that will be available to all workers. 15:22:23 +1 15:22:32 +1 for calling it 'script' no need to be a 'shell script'. 15:22:45 The command to create the signing service will return an HREF for the newly created signing service. This reference can then be used to assign that service to a specific repository. 15:22:51 x9c4: yep, shell was just me trying to diferentiate it from relic itself 15:23:04 x9c4: for anything more complicated, python or such is better 15:23:34 The repo admin is going to assign the signing service to a specific repository. 15:24:16 Is this in the plugin domain? 15:24:22 What kind of REST operations do you need on signing service(s)? enumerate signing services, list keys available for a signing service? 15:24:29 x9c4: no, this would be pulp core 15:24:36 fine. 15:25:16 List signing services or get a specific one. Not sure what kind of details would be made available though. 15:25:17 as a repo admin I may want to know what signing keys were configured in a service, so I can pick the one I want (although typically it may be one-to-one) 15:26:06 yeah ... i guess a list of key ids would be good to expose via REST api 15:26:26 And it should have a name, as we probably don't want to expose the path_to_script. 15:26:33 +1 15:26:44 dkliban: for this, I was proposing that the executable can be invoked with --list-keys 15:27:04 dkliban: so the executable would have two interfaces: --sign {src} {dst} and --list-keys 15:27:33 that's reasonable 15:27:56 our hhour is almost up 15:28:28 i am going to write a summary of today's discussion in the document. i also try to write up some stories in pulp.plan.io 15:29:18 i am hoping we can continue our planning of this feature in pulp.plan.io ... though if we want to meet again, i am open to that also 15:29:20 Thank you! 15:29:46 thank you everyone for participating today! 15:30:04 can we also clarify some details about scritpt parameters and env vars? 15:30:12 I mean in the summary or in the redmine tickets 15:30:27 and the reasoning behind choosing a certain way 15:30:48 yeah ... we will 15:30:52 #endmeeting 15:30:52 !end