16:00:41 <dkliban> #startmeeting Pulp Triage 2019-06-19 meeting
16:00:41 <dkliban> #info dkliban has joined triage
16:00:41 <dkliban> !start meeting
16:00:41 <pulpbot> Meeting started Wed Jun 19 16:00:41 2019 UTC.  The chair is dkliban. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:41 <pulpbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:00:41 <pulpbot> The meeting name has been set to 'pulp_triage_2019-06-19_meeting'
16:00:41 <pulpbot> dkliban: dkliban has joined triage
16:00:54 <ttereshc> #info ttereshc has joined triage
16:00:54 <ttereshc> !here
16:00:54 <pulpbot> ttereshc: ttereshc has joined triage
16:00:57 <dkliban> !topic Pulp 2 to 3 migrations
16:00:57 <pulpbot> dkliban: (topic [<channel>]) -- Returns the topic for <channel>. <channel> is only necessary if the message isn't sent in the channel itself.
16:01:20 <jsherrill> #info jsherrill has joined triage
16:01:20 <jsherrill> !here
16:01:20 <pulpbot> jsherrill: jsherrill has joined triage
16:01:22 <dkliban> !topic "Pulp 2 to 3 migrations"
16:01:22 <pulpbot> dkliban: (topic [<channel>]) -- Returns the topic for <channel>. <channel> is only necessary if the message isn't sent in the channel itself.
16:01:25 <dkliban> oh well
16:01:28 <dkliban> !here
16:02:13 <dkliban> bmbouter ?
16:03:48 <ttereshc> dkliban, how about we leave time for hardlinks/no hardlinks topic, even if we are not done with all the use cases? it seems like a blocker for our progress at the moment
16:03:58 <dkliban> let's start with that
16:04:10 <dkliban> jsherrill: and everyone else: ...
16:04:19 <dkliban> last week we discussed not using hardlinks to migrate content
16:04:37 <dkliban> instead the migration task is simply going to record the current path of a content unit as the path for an Artifact in pulp
16:04:53 <dkliban> this way we just have to create database entries to migrate the artifacts
16:05:25 <dkliban> after all the content is migrated the user will have an opportunity to run a separate task that will move the artifacts into their proper pulp 3 storage location
16:05:57 <dkliban> and that final task can do it in such a way that the file exists in 2 location for some period of tim e
16:06:05 <dkliban> the database is updated with the new location of the file
16:06:10 <dkliban> then the old file is removed
16:06:25 <bmbouter> #info bmbouter has joined triage
16:06:25 <bmbouter> !here
16:06:25 <pulpbot> bmbouter: bmbouter has joined triage
16:06:27 <dkliban> this can be performed in small batches so that very little extra storage is needed
16:06:44 <jsherrill> that seems resonable to me
16:06:53 <ttereshc> dkliban, just an opportunity or should we enforce/insist somehow? otherwise there will be always installations with pulp2 paths :(
16:07:10 <dkliban> we should require it probably
16:07:21 <bmbouter> I hope we do, it's been a multi-year issue and we could resolve it with this
16:07:35 <jsherrill> require it at which point?
16:07:48 <dkliban> jsherrill: that's the hard part ...
16:08:11 <dkliban> jsherrill: i am not sure how we could require it other than not let you serve artifacts that are not in the artifacts directory
16:08:29 <dkliban> and have pulp log that it's not serving the content because it's in the wrong place
16:08:37 <jsherrill> but that would be some future release right?
16:08:37 <dkliban> log it loudly
16:08:46 <bmbouter> I imagined this would run at the end of the MP
16:08:58 <bmbouter> a followup task that would just auto-run
16:09:02 <dkliban> yeah .... this would ship in the same package that bring the migration task
16:09:11 <bmbouter> and it's safe to cancel if it's taking too long because it resumes safely
16:09:12 <jsherrill> but this would remove it in the old location?
16:09:30 <bmbouter> it would after fully creating it in the new location and updating the pulp2 database
16:09:31 <dkliban> jsherrill: i see your concern
16:09:53 <jsherrill> if it does, then it moves a long running part of the migration to 'switch over'
16:09:56 <bmbouter> and it shoiuld be safe because the content exists in both the old and new location until everything uses the new location
16:09:58 <dkliban> jsherrill: is concerned that if we remove it in pulp 2, we need to have it served by pulp 3 already
16:09:59 <jsherrill> that we are trying to make it as short as possible
16:10:09 <bmbouter> oh I'm not advocating removing it
16:10:12 <bmbouter> but moving it
16:10:19 <bmbouter> pulp2 is still online in this example
16:10:29 <ttereshc> bmbouter, what if user uses S3 storage in pulp3
16:10:55 <jsherrill> then we're at 2x disk space, and slowness of the copy
16:11:00 <jsherrill> whats teh advantage ?
16:11:02 <jsherrill> the*
16:11:21 <ttereshc> jsherrill, if it's the same fs, it can be just hard links
16:11:24 <bmbouter> yup
16:11:44 <bmbouter> and the batches mean even if hardlinks couldn't be used it would only be 2x of the batch size
16:11:57 <bmbouter> some filesystems may just not support hardlinking and it probably needs a fallback mode
16:11:59 <bmbouter> perhaps
16:12:11 <bmbouter> but I think we can assume hardlinking is fine for now until it isn't and then implement it
16:12:14 <jsherrill> so this isn't a discussion about 'no hardlinks', its a discussion about when hardlinks are possible?
16:12:22 <jsherrill> aren't*
16:12:33 * bmbouter came in halfway
16:13:05 <jsherrill> it sounded like it was starting as a discussion to not use hardlinks, and use a different way
16:13:41 <dkliban> i guess so
16:13:56 <ttereshc> jsherrill, I see what you mean. I think it's more a discussion about the migration process itself. And we went directly into post-migration steps
16:14:13 <bmbouter> what does post-migration mean?
16:14:18 <ttereshc> and they can potentially have hardlink or moving files around
16:14:46 <bmbouter> yeah I think either mechanism is fine, that's almost just a detail the implementnation will deal with. in both cases we need to be sure it's totally safe to pulp2 to remove the old location
16:14:57 <bmbouter> I don't mean to glossover it
16:15:03 <dkliban> that is a good question bmbouter. can users operate Pulp 3 without creating artifacts in their proper location
16:15:14 <jsherrill> i think removal from pulp2 should probably not be done in the migration tooling at all
16:15:23 <jsherrill> and left up to the user?
16:15:24 <dkliban> correct
16:16:02 <bmbouter> that would be fine w/ me since it solves the problem
16:16:13 <bmbouter> it will 2x the disk usage though and that was one of our design goals
16:16:24 <dkliban> bmbouter: only if it's a different file system
16:16:32 <jsherrill> or if you can't use hardlinks
16:16:32 <dkliban> hard links don't take up that much space
16:16:38 <dkliban> yeah
16:16:44 <bmbouter> this sounds great hardlinks to save the day
16:16:50 <dkliban> lol
16:17:14 <dkliban> ttereshc: what do you think about keeping the removal of pulp 2 content units up to the user?
16:17:16 <bmbouter> also this solves the S3 case b/c the pulp2 content stays in place and the pulp3 content goes to S3
16:17:38 <ttereshc> dkliban, I'm +1 for not touching pulp2 at all
16:17:43 <dkliban> great
16:17:55 <ttereshc> bmbouter, exactly
16:18:03 <dkliban> so to summarize ...
16:19:10 <dkliban> The migration task is going to migrate artifacts by recording their current path in pulp 2. After this task finishes, another task is going to create hard links or copy files into their proper pulp 3 location.
16:19:50 <dkliban> this other task could possibly run in parallel
16:20:12 <jsherrill> whats the benefit of making it  separate ?
16:20:19 <ttereshc> oh, so the other task is a part of migration process?
16:20:45 <ttereshc> I thought it can be done slowly in the background with the operational pulp3
16:20:56 <dkliban> that's the question i want to answer: can users of pulp 3 operate it if the artifacts are still in their pulp 2 location?
16:21:13 <dkliban> i want to say yes, but bmbouter was not as excited about that
16:21:22 <dkliban> did i understand that correctly bmbouter?
16:22:44 <dkliban> jsherrill: making it separate would allow users of pulp 3 to start operating sooner. that is if we don't require them to move the files into place first.
16:23:03 <jsherrill> right, that makes sense, if we're not going to let them operate sooner, it doesn't make a ton of sense
16:23:10 <dkliban> i agree
16:23:13 <ttereshc> +1
16:24:01 <ttereshc> if it's ok to operate pulp3 with pulp2 paths, is there a way to force to move content to pulp3 before the next upgrade?
16:24:22 <dkliban> yeah ... we could build that into pulp 3.1
16:24:42 <jsherrill> i'm not sure i see a ton of value tbh
16:24:48 <jsherrill> this migration is going to take a long time
16:24:57 <jsherrill> does it matter to me if it takes 5 hours versus 10 hours?
16:25:00 <jsherrill> not really
16:25:24 <ttereshc> which part of migration? moving files around? or the db operations?
16:25:45 <jsherrill> whatever part it is before i turn on pulp3
16:26:15 <jsherrill> is being able to start using pulp3 sooner worth the complexity of having this 2-tiered migration?
16:26:16 <bmbouter> it's important to me that the filesystem layout of pulp3 be in it's content addressable storage like it's designed
16:26:55 <bmbouter> I'm uncomfortable w/ 3.0 not having ^
16:27:14 <jsherrill> i think thats how i lean too
16:27:18 <jsherrill> keep it simple when possible
16:27:28 <jsherrill> and this is just adding complexity for not a ton of value IMO
16:27:36 <dkliban> i agree
16:27:52 <bmbouter> we never fixed it because we never could reasonably. now we can!
16:28:04 <dkliban> so for users that don'thave a filesystem that supports hardlinks we will need to recommend thathtye double their storage
16:28:14 <jsherrill> for sure
16:28:17 <bmbouter> yup
16:28:21 <bmbouter> or switch to a filesystem w/ hardlinks
16:28:25 <bmbouter> that is definitly an option for them too
16:28:26 <dkliban> cool ... and then we will do this as one task
16:28:35 <ttereshc> my concerns are - there is always this final part of migration (and if moving files around is involved, it can be long) + diskspace if hardlinks are not supported
16:29:16 <bmbouter> I hear that. in that case, the pulp2 system would have to be producing content at a pretty staggering rate
16:29:37 <bmbouter> at some point the runtime of checking over the MP and running it, relative to the rate of new incoming content will be a race
16:30:16 <bmbouter> and to the extend that new in-bound content rate >> MP completion rate there will be downtime
16:30:32 <dkliban> yep
16:30:36 <dkliban> downtime for the workers
16:30:39 <ttereshc> we never know when customer decides to make a final part and there are who re-sync 100s or repos daily, so we need to hope that it's not done on a day when they decide to sync new Fedora
16:30:39 <bmbouter> the user could decide to reduce the inbound pulp2 content rate, perhaps take pulp2's tasking offline, do fewer sync's
16:30:48 <bmbouter> yeah exactly
16:31:01 <bmbouter> so this gets to a point that we should consider at some point
16:31:32 <bmbouter> which is estimation time to completion
16:31:37 <bmbouter> at some future meeting/time
16:32:18 <bmbouter> when the user is doing the final cutover they will be wondering "when's it finishing" so progress reporting in that tasking is important
16:32:30 <dkliban> yep
16:32:41 <ttereshc> yeah, we should provide some confidence that the last part wouldn't take forever
16:33:31 <bmbouter> I think the best thing we can do is provide clear reporting on how much still needs to be done for this MP
16:33:40 <dkliban> yep
16:33:41 <ttereshc> yes
16:33:45 <ttereshc> I meant that
16:33:59 <dkliban> cool ... so to summarize ...
16:34:00 <ttereshc> I think we can move on, it seems like we are in agreement
16:34:01 <bmbouter> and in the even they shut off their pulp2 system (to do the final cutover) the progress reporting on that one will tell the tale
16:34:03 <ttereshc> :)
16:35:20 <dkliban> Pulp 3's migration task will migrate content by either creating a hard link in the pulp 3 artifact storage location or by copying it there. The migration task will report how much content has been migrated and how much is left.
16:35:49 <bmbouter> that reads well to me
16:36:12 <dkliban> cool. i hope we can go to our next topic now
16:36:20 <ttereshc> +1 we'll figure out details for reporting later
16:36:26 <dkliban> yep
16:37:01 <dkliban> ttereshc: what question did we have for jsherril last week with regard to distributors ?
16:37:17 <ttereshc> which details we need to migrate I think
16:37:17 <dkliban> i think it was about the ssl certs and such
16:37:30 <ttereshc> protected repos
16:37:44 <dkliban> yeah ... jsherrill: for protected repos, what in pulp 2 needs to be migrated into pulp 3
16:37:53 <dkliban> i think we need to create some content guards
16:37:56 <bmbouter> yup
16:38:01 <jsherrill> currently we use global settings for protected repos
16:38:52 <dkliban> jsherrill: where are the global settings stored? in the /etc/pulp ?
16:39:03 <jsherrill> i think either, we'd need to indicate which distributors need a guard, or handle that ourselves afterwards
16:39:05 <jsherrill> dkliban: yes
16:39:19 <jsherrill> /etc/pulp/repo_auth.conf  i think?
16:39:20 <bmbouter> I think the broader user base of pulp would enjoy the former
16:39:39 <ttereshc> I agree
16:39:48 <dkliban> bmbouter: yes, but i am struggling to see how to specify the details for that guard
16:40:01 <dkliban> bmbouter: would the user specify a path to a cert in /var/lib/pulp ?
16:40:02 <bmbouter> me too. I'm looking in the pulp2 code now
16:40:52 <bmbouter> https://github.com/pulp/pulp/blob/2-master/oid_validation/pulp/oid_validation/oid_validation.py#L94-L103
16:41:04 <bmbouter> jsherrill: this is akin to the OidValidator right?
16:41:31 <jsherrill> bmbouter: yeah
16:42:09 * bmbouter is digging
16:42:40 <bmbouter> https://github.com/pulp/pulp/blob/2-master/repoauth/pulp/repoauth/repo_cert_utils.py#L120-L135
16:42:46 <dkliban> i think the migration task could look up these global settings and create a content guard with the,m
16:42:51 <bmbouter> agreed
16:42:54 <bmbouter> it's all right here
16:43:50 <dkliban> so we need to be able to support 2 use cases: one where the global repo protection exists and another one where it is specifically defined on a distributor
16:44:04 <jsherrill> dkliban: well we don't want global protection
16:44:04 <bmbouter> +1
16:44:17 <jsherrill> we want to move to per-repo protection
16:44:20 <bmbouter> right but I think the MP that Pulp2 produces needs to have that in there
16:44:28 <jsherrill> ahh k
16:44:32 <bmbouter> and then you'll need to adjust the MP to move to per-repo
16:44:35 <dkliban> it needs to convert the global protection to per repo
16:44:47 * bmbouter waves hands during syntax questions
16:44:54 <dkliban> we don't have global protection in pulp 3
16:45:07 <dkliban> so it's implied that it will be per repo in pulp 3
16:45:14 <bmbouter> good point
16:45:35 <bmbouter> so pulp2 would produce the MP that would already be per-repo
16:45:42 <dkliban> yes
16:45:49 <dkliban> but i don't know how that would look
16:45:50 <bmbouter> the only ambiguous case is where global and per-distributor is used
16:46:01 <dkliban> per distributor wins
16:46:07 <bmbouter> good call
16:46:11 <bmbouter> you all agree?
16:46:32 <dkliban> we need to come up with how to describe this in the MP
16:46:43 <bmbouter> can we do it as a post-meeting todo
16:46:45 <bmbouter> and I can help
16:46:47 <ttereshc> if there is a global protection configured in pulp2, do we add it for every repo in pulp3?
16:46:59 <bmbouter> I believe the MP would default to that, and the user can adjust
16:47:27 <dkliban> ttereshc: yes
16:47:35 <dkliban> for RPM only though
16:47:47 <dkliban> or does pulp 2 apply it to all repos?
16:47:58 <bmbouter> all repos
16:48:05 <bmbouter> it's done in the webserver generically
16:48:06 <ttereshc> does it make sense to add this option to repo_versions dictionary in the MP?
16:48:11 <ttereshc> see L323
16:49:55 <dkliban> ttereshc: yeah
16:49:55 <bmbouter> that looks good to me
16:49:58 <ttereshc> or what is the way to customize the cert_guards creation at migration time
16:50:00 <ttereshc> kk
16:50:34 <bmbouter> I don't think we can customize at migraiton additionally
16:50:41 <bmbouter> just migrate or not
16:50:49 <bmbouter> what do you all think?
16:50:53 <ttereshc> bmbouter, sorry, I meant selectively migrate
16:51:16 <ttereshc> for this repo migrate and for this not
16:51:44 <dkliban> yeah ... i think having an explicit field on a repository version is the way to do it
16:52:29 <bmbouter> ttereshc: the L323 example allows for selective migration I think. is that right?
16:52:41 <dkliban> yes
16:52:46 <ttereshc> yes
16:53:04 <ttereshc> do we require it in order to create cert_guards?
16:53:29 <bmbouter> I think so and the MP includes it if pulp2 had that config
16:53:31 <ttereshc> or do we need also a top-level field - create cert_guards for all
16:53:51 <dkliban> yeah
16:53:57 <dkliban> i think a top level one should also exist
16:54:17 <dkliban> and it will tell the migration task to use the global settings to create content guard to use with tall
16:54:34 <dkliban> or should it create a copy of the same content guard for each?
16:54:55 <bmbouter> I agree w/ a top level
16:55:09 <bmbouter> and then how you apply those created objects to repositories is another expression in the MP
16:55:23 <bmbouter> and they can match on ContentGuard.name or ContentGuard.id
16:55:41 <bmbouter> btw ContentGuards have id, name, descrpition, and ca_certificate
16:56:21 <bmbouter> I think the MP top-level and associating it w/ a repo insid ethe repo_version config would refer to the ContentGuard by 'name'
16:56:32 <dkliban> is that name unique?
16:56:48 <bmbouter> since 'id' is selected by Django
16:56:51 <bmbouter> yes it's unique (I just looked)
16:57:02 <dkliban> coool beans
16:57:51 <ttereshc> why are multiple certguard needed if they will be the same? I
16:58:13 <dkliban> i don't think multiple are needed
16:58:21 <dkliban> but i was just seeing what jsherrill had in mind
16:58:27 <ttereshc> ok cool
16:58:51 <bmbouter> I don't see a clear use case for multiple
16:59:01 <bmbouter> specifically when they contain the same ca_certificate data...
16:59:05 <dkliban> we are almost at the end of the hour
16:59:12 <jsherrill> yeah, we'd just need 1
16:59:30 <dkliban> ttereshc: are you going to update the etherpad with an example for the top level cert guard specification/
16:59:38 <ttereshc> dkliban, sure
16:59:56 <dkliban> thank you everyone for participating today
17:00:03 <dkliban> thank you ttereshc for organizing
17:00:12 <dkliban> #endmeeting