15:30:08 <fao89> #startmeeting Pulp Triage 2021-01-22 15:30:08 <fao89> !start 15:30:08 <fao89> #info fao89 has joined triage 15:30:08 <pulpbot> Meeting started Fri Jan 22 15:30:08 2021 UTC. The chair is fao89. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:30:08 <pulpbot> Useful Commands: #action #agreed #help #info #idea #link #topic. 15:30:08 <pulpbot> The meeting name has been set to 'pulp_triage_2021-01-22' 15:30:08 <pulpbot> fao89: fao89 has joined triage 15:30:29 <fao89> Open Floor! 15:30:35 <daviddavis> #info daviddavis has joined triage 15:30:35 <daviddavis> !here 15:30:35 <pulpbot> daviddavis: daviddavis has joined triage 15:30:37 <ppicka> #info ppicka has joined triage 15:30:37 <ppicka> !here 15:30:37 <pulpbot> ppicka: ppicka has joined triage 15:30:40 <ggainey> #info ggainey has joined triage 15:30:40 <ggainey> !here 15:30:40 <pulpbot> ggainey: ggainey has joined triage 15:31:03 <fao89> topic: Spam script removing actual users 15:31:12 <bmbouter> #info bmbouter has joined triage 15:31:12 <bmbouter> !here 15:31:12 <pulpbot> bmbouter: bmbouter has joined triage 15:31:28 <daviddavis> where were we with this on tuesday 15:31:59 <fao89> I don't remember, but it was a pretty intense conversation 15:32:03 <daviddavis> lol 15:32:14 <daviddavis> we were talking about making potential spam content private 15:32:34 <bmbouter> oh yeah and I was saying that goes against a long standing goal motivated by many problems 15:32:36 <dkliban> #info dkliban has joined triage 15:32:36 <dkliban> !here 15:32:36 <pulpbot> dkliban: dkliban has joined triage 15:32:37 <gerrod> #info gerrod has joined triage 15:32:37 <gerrod> !here 15:32:37 <pulpbot> gerrod: gerrod has joined triage 15:32:44 <fao89> oh yeah, and bmbouter was thinking about not allowing private content 15:32:49 <ttereshc> #info ttereshc has joined triage 15:32:49 <ttereshc> !here 15:32:49 <pulpbot> ttereshc: ttereshc has joined triage 15:33:02 <ggainey> heh - yeah, I think we'd gotten to "we don't remove users anymore, we should prob be making content private, but we want to get rid of the private-button altogether" 15:33:04 <dkliban> yeah ... i was going to file a task to make the script mark things private ... but i didn't do that 15:33:58 <bmbouter> yeah I don't want to trade one problem for a new problem 15:34:05 <bmbouter> so what is the root cause of our current problem? 15:34:08 <x9c4> #info x9c4 has joined triage 15:34:08 <x9c4> !here 15:34:08 <pulpbot> x9c4: x9c4 has joined triage 15:34:17 <daviddavis> valid content is being removed as spam 15:34:29 <ggainey> "spam filtering is hard and can have false positives" 15:34:57 <fao89> I think we are just trusting on CI code from pulp-ci, spam and bugzilla jobs 15:35:34 <fao89> we deployed the code and we are just praying for it to be right 15:36:15 <gerrod> is there still spam on GH issues? aren't we looking to switch anyway? 15:37:54 <ggainey> if we're worried about flase-positives, we could just turn off the spam-code, and everyone take onboard a atask of "check faily activity dashboard on plan.io and hand-remove obvious spam" 15:38:02 <ggainey> (which is what we were doing before) 15:38:09 <ggainey> ^flase^false 15:38:38 <ggainey> esp if we're thinking of migrating to using GH "soon"(ish) 15:38:45 <fao89> I'm seeing less spam, maybe because of the script? Can we disable it and see if we are having less spam even without the script? 15:38:52 <bmbouter> we are looking to switch 15:38:54 <ttereshc> gerrod, we are planning to, just we need to do something before we've moved 15:38:57 <x9c4> Can we change the script to flag spam? 15:38:58 <ggainey> sounds like a useful experiment 15:39:10 <fao89> https://github.com/pulp/pulp-ci/actions 15:39:13 <daviddavis> how do we flag spam? 15:39:20 <ggainey> (turning it off for a bit to see how bad things actually are right now, that is) 15:39:40 <x9c4> We add a label and then it is easier to review what needs to be deleted. 15:39:44 <bmbouter> redmine doesn't have a flag facility and adding more custom fields takes us away from the goal of switching to GHI 15:39:46 <dalley> the spam still happens even with it turned on, isn't it only deleted after a week or something 15:39:59 <ggainey> we run once/wk iirc 15:40:25 <bmbouter> the question I keep asking is: is this a severe enough issue for us to take an action 15:40:46 <bmbouter> my conclusion to that atm is no even if someusers content got deleted because in fairness they didn't follow the process they were emailed about 15:40:51 <daviddavis> hard to tell because we don't know how much content has actually been removed 15:41:16 <daviddavis> if we thinking about doing nothing, I would maybe just disable the spam script 15:41:28 <ggainey> daviddavis: at some point, "sever" is defined by "how many ppl are yelling at us" :) 15:42:27 <ggainey> but I'd be OK w/disabling the script and seeing how bad the problem is right now - as long as we as a team agree that looking for and removing obvious spam "by hand' is something we'll do semi-regularly 15:42:43 <fao89> https://github.com/pulp/pulp-ci/blob/master/.github/spam_cleaner.py 15:43:08 <daviddavis> one of my concerns is that no one is monitoring or maintaining this script 15:43:20 <fao89> +1 15:43:27 <bmbouter> this is also my main concer 15:43:29 <bmbouter> concern 15:43:47 <daviddavis> if someone were, I'd be less concerned about keeping it running 15:44:23 <fao89> we have a code that interact with users and we don't monitor it, so I vote to disable it 15:44:51 <bmbouter> that makes sense to me also 15:45:11 <bmbouter> also the spam isn't killing us but having someone monitor spam takes away from productive work 15:45:13 <ggainey> I'm good with it - it was a good experiment, but see prev RE "spam filtering is hard" 15:45:23 <dkliban> yep 15:45:46 <dkliban> should we go ahead and disable it today? 15:45:50 <ggainey> so let's turn it off and see if we're overwhelmed - may not be a problem, the spam-ecosystem is always changing 15:45:53 <ggainey> yes 15:46:03 <ggainey> or rather, "I am in favor" :) 15:46:12 <x9c4> +1 15:46:24 <bmbouter> +1 15:46:28 <daviddavis> +1 15:47:05 <dkliban> +1 15:47:14 <dkliban> i'll make a PR to disable it now 15:47:27 <fao89> dkliban++ 15:47:27 <pulpbot> fao89: dkliban's karma is now 581 15:47:29 * ggainey cheers wildly (but not spammily) 15:47:32 <bmbouter> ty 15:47:37 <dkliban> lol 15:47:39 <fao89> next topic: New tags in Redmine 15:47:40 <daviddavis> excellentayyyy 15:48:37 <fao89> who can speak bout the new tags? Is it just FYI? 15:48:44 <daviddavis> I believe dalley added it 15:49:00 <x9c4> bmbouter, will you bring up tags to delete again if we add new ones? 15:49:21 <bmbouter> like when we switch to GHI (github issues?) 15:49:27 <bmbouter> if so, no because those tags will become labels 15:49:39 <bmbouter> I still don't think we're ready to capture pulp.next things though 15:49:51 <bmbouter> we did this in pulp2 and spent effort on planning pulp3 only to get into pulp3 and throw all of it out 15:50:31 <ttereshc> bmbouter, this came up at backlog grooming 15:50:53 <ttereshc> there are some really good issues/stories,very well written but we know we won't get to them\ 15:51:23 <ttereshc> they are up to date so far 15:51:42 <ttereshc> I agree with not planning pulp.next too early 15:51:46 <bmbouter> maybe keeping this is exactly what we should do 15:52:04 <bmbouter> but we were just saying that the longer the time between planning and implementation the more that planning becomes wasted 15:52:09 <fao89> Pulp-NG 15:52:17 <ttereshc> bmbouter, I agree 15:52:36 <bmbouter> and we may loose what we already have 15:52:38 <ttereshc> I'm on the fence, just a pity to throw away some of them 15:52:49 <bmbouter> but by keeping it I'm concerned we'll continue to build it out and throw good time after bad 15:52:55 <bmbouter> throw away 15:53:08 <bmbouter> fyi it's not a big deal for me, this is just my opinion 15:53:09 <daviddavis> I still don't understand the hesitation with closing issues personally. it's not like we're deleting them 15:53:20 <bmbouter> I feel the same 15:53:48 <ttereshc> how many times have you re-opened the story or issue instead of wrting a new one? 15:53:49 <ggainey> if we close 'pulp next' issues with a standard comment, we can find them all via search, yeah? 15:54:25 <daviddavis> probably the same number of times I've filed a dupe issue for an already open one 15:54:27 <ttereshc> I don't have strong opinion here, just sharing a sentiment I noticed at backlog grooming 15:54:52 <ttereshc> dalley, any thoughts? I believe you are in favor of not closing 15:54:55 <daviddavis> yea, I don't have a strong opinion either 15:57:29 <daviddavis> maybe return to this when dalley is present? 15:57:50 <bmbouter> +1 15:58:20 <ttereshc> +1 15:58:30 <ggainey> +1 15:58:35 <fao89> next topic: Make worker ttl configurable 15:58:50 <dalley> as long as we tag them so that we can find them later to re-open them, I'm OK with closing them 15:59:10 <bmbouter> we could make a custom query w/ the issue numbers 15:59:11 <dalley> I'm against closing important / useful features or issues without a way to dig them back up 15:59:30 <bmbouter> that way we don't take on another thing to migrate to GHI and yet we don't loose them 16:00:01 <bmbouter> just an idea 16:01:01 <ttereshc> I can speak to the next topic 16:01:35 <bmbouter> I can make a query to demo it and we can leave the pulp.next issues open for now 16:01:48 <bmbouter> if someone can put on the next open floor to revisit 16:01:59 <fao89> I put it 16:02:13 <bmbouter> that'll work let's go into next yes? 16:02:22 <ggainey> +1 16:02:32 <daviddavis> +1 16:02:37 <dalley> +1 16:02:41 <fao89> last topic: Make worker ttl configurable 16:02:49 <ttereshc> Make the hearbeat for workers configurable is a katello ask and it's because of the issue with io when customers have hdd's. 16:03:03 <bmbouter> ohhhhh 16:03:15 <ttereshc> and it's noticeable when running the migration plugin. 16:03:31 <bmbouter> yeah so this comes up maybe once a month and I usually say it's because we're doing something wrong elsewhere and the db is blocking in an unhealthy way 16:03:37 <ttereshc> it's likely that ther are things to improve in plugin itself 16:03:56 <ttereshc> but it's quite easilty reproducible if you switch from ssd to hdd 16:04:00 <bmbouter> if we're in a situation where a single write can't occur for 30 seconds we have a bigger problem 16:04:41 <ttereshc> bmbouter, current dogfood server doesn't use very modern hardware and the hearbeat for pulp2 was increased there for that reason 16:05:10 <bmbouter> yes but this is signalling a deeper problem 16:05:12 <ttereshc> folks say that it reflects a situation which some customers are often in 16:05:19 <bmbouter> I agree they are in that situation 16:05:33 <bmbouter> but I'm trying to shift the convo to investigating the root cause not the symptom 16:05:36 <dalley> the problem is that there's just a huge amount of IO on a slow disk and no way to prioritize certain activities over others 16:05:55 <bmbouter> then we shouldn't be driving the db that hard or have a backpressure monitor 16:06:05 <bmbouter> consider that other services run on that postgres server too 16:06:12 <ttereshc> so we are looking into improving migration plugin and track what causes that, we already fixed a lot 16:06:12 <dalley> bmbouter, it's not just the DB, it's file ops too 16:06:31 <bmbouter> if we're DoSing the db and the filesystem we are DoSing more than just pulp's heartbeats 16:06:54 <ttereshc> I think those can go in parallel, make it configurable and figure out deeper problems and if it's possible to fix them 16:07:31 <bmbouter> I'm not confident we will address the real problem if we do that 16:07:42 <ttereshc> if we do what? 16:07:49 <ttereshc> go into both directions? 16:07:54 <bmbouter> yes 16:08:04 <bmbouter> I'm uncomfortable with that option at least 16:08:23 <ttereshc> maybe we should have a convo with katello folks on pulp-dev 16:08:41 <bmbouter> I'm into that, I really really want to make it usable for them 16:08:45 <ggainey> +1 16:08:57 <ttereshc> it's for customers on 3.18 16:09:14 <ttereshc> pulpcore 3.7 16:09:27 <bmbouter> yup and we've known about this for a long time 16:09:50 <ttereshc> I think we can move on to the triage and I'll bring it up on pulp-dev, wdyt? 16:09:58 <bmbouter> that sounds great 16:10:02 <daviddavis> +1 16:10:19 <fao89> !next 16:10:21 <fao89> #topic https://pulp.plan.io/issues/8131 16:10:21 <pulpbot> fao89: 7 issues left to triage: 8131, 8122, 8116, 8115, 8112, 7950, 7922 16:10:22 <pulpbot> RM 8131 - mcorr - POST - Add links to pulp plugin docs into the body of docs.pulpproject.org 16:10:23 <pulpbot> https://pulp.plan.io/issues/8131 16:10:50 <fao89> #idea Proposed for #8131: convert to task 16:10:50 <fao89> !propose other convert to task 16:10:50 <pulpbot> fao89: Proposed for #8131: convert to task 16:11:04 <daviddavis> +1 16:11:05 <ggainey> +1 16:11:09 <bmbouter> +1 16:11:10 <ppicka> +1 16:11:11 <fao89> #agreed convert to task 16:11:11 <fao89> !accept 16:11:11 <pulpbot> fao89: Current proposal accepted: convert to task 16:11:11 <ttereshc> it's in POST 16:11:12 <fao89> #topic https://pulp.plan.io/issues/8122 16:11:12 <pulpbot> fao89: 6 issues left to triage: 8122, 8116, 8115, 8112, 7950, 7922 16:11:13 <pulpbot> RM 8122 - iballou - NEW - Backport Request: 8099 (File upload causes django.security.SuspiciousFileOperation:ERROR) to 3.9 16:11:14 <pulpbot> https://pulp.plan.io/issues/8122 16:11:17 <ttereshc> are we adding such to the sprint? 16:12:51 <bmbouter> for 8131 I'm ok w/ that 16:12:58 <bmbouter> mcorr do you plan to take as assigned? ^ 16:13:44 <mcorr> I raised a PR for that bmbouter - I am sorry for making trouble 16:13:54 <bmbouter> mcorr: no trouble! 16:14:02 <bmbouter> got a link? 16:14:11 <mcorr> sure one sec 16:14:28 <fao89> https://github.com/pulp/pulpcore/pull/1079 16:14:38 <mcorr> https://github.com/pulp/pulpcore/pull/1079 bmbouter 16:14:42 <mcorr> aah thanks fao89 16:14:52 <fao89> pulpbot did add the link to the issue 16:14:52 <pulpbot> fao89: Error: "did" is not a valid command. 16:15:00 <bmbouter> so yeah add to sprint plz 16:15:09 <fao89> about the backport should I accept and add? 16:15:22 <daviddavis> I think it's done already 16:15:22 <bmbouter> I think so but dkliban would know ^? 16:15:23 <daviddavis> dkliban: ? 16:15:25 <bmbouter> yup 16:15:33 <ttereshc> I think it's not a backport, since thre is no 3.10 16:15:44 <daviddavis> it says to 3.9 16:16:02 <ttereshc> yeah, so it's a normal Z stream, no? 16:16:07 <fao89> yeah, it is just z-stream 16:16:16 <ttereshc> there is no release to backport it from 16:16:25 <daviddavis> I thought any issue to a z-stream was a backport 16:16:57 <ttereshc> I think we can have a fix with a db migration in a z stream 16:17:01 <ttereshc> if it's a latest one 16:17:09 <ttereshc> but we can't backport such 16:17:24 <daviddavis> I don't think we need a db migration for https://pulp.plan.io/issues/8122 16:17:43 <ttereshc> I'm just saying in theory 16:17:56 <ttereshc> the diff between backport and the latest z stream 16:18:14 <ttereshc> anyway, I'm +1 to close 8122 16:18:17 <ttereshc> as a dupe 16:18:19 <daviddavis> so to me, backport means taking something from master or a later release and applying to a release branch and releasing 16:19:06 <fao89> #idea Proposed for #8122: close and point the link to z-stream 16:19:06 <fao89> !propose other close and point the link to z-stream 16:19:06 <pulpbot> fao89: Proposed for #8122: close and point the link to z-stream 16:19:12 <ggainey> +1 16:20:01 <fao89> #agreed close and point the link to z-stream 16:20:01 <fao89> !accept 16:20:01 <pulpbot> fao89: Current proposal accepted: close and point the link to z-stream 16:20:02 <fao89> #topic https://pulp.plan.io/issues/8116 16:20:02 <pulpbot> fao89: 5 issues left to triage: 8116, 8115, 8112, 7950, 7922 16:20:03 <pulpbot> RM 8116 - paji@redhat.com - NEW - Export is generating the incorrect 'next_version' causing import to fail. 16:20:04 <pulpbot> https://pulp.plan.io/issues/8116 16:20:05 <dkliban> re 8122: it should be closed 16:20:07 <ttereshc> daviddavis, my understanding excludes the master branch if it goes to the latest Z, but I can see your logic as well 16:20:23 <ggainey> 8116 - accept-and-add please - we need to fix this 16:20:37 <fao89> #idea Proposed for #8116: accept and add to sprint 16:20:37 <fao89> !propose other accept and add to sprint 16:20:37 <pulpbot> fao89: Proposed for #8116: accept and add to sprint 16:20:59 <daviddavis> I'm not sure how the workflow is to create these other-things-which-are-not-backports. do users still file backports? 16:21:33 <ggainey> gang, can we move the bakports-discussion to some othe rtime? we have a lot of issues to get thru 16:21:37 <ttereshc> they just ask to release a z stream I think and include that fix 16:21:41 <fao89> I think we can add backport to the next open floor 16:21:45 <ggainey> +1 16:21:57 <daviddavis> yea we discuss this more 16:22:04 <ttereshc> whatever works for everyone I'm fine with 16:22:57 <fao89> 8116 anyone agains accepting and adding to sprint? 16:23:08 <daviddavis> nope, works for me 16:23:14 <bmbouter> wfm 16:23:19 <fao89> #agreed accept and add to sprint 16:23:19 <fao89> !accept 16:23:21 <pulpbot> fao89: Current proposal accepted: accept and add to sprint 16:23:22 <fao89> #topic https://pulp.plan.io/issues/8115 16:23:22 <pulpbot> fao89: 4 issues left to triage: 8115, 8112, 7950, 7922 16:23:23 <pulpbot> RM 8115 - bmbouter - NEW - Satellite issue sync is broken 16:23:24 <pulpbot> https://pulp.plan.io/issues/8115 16:24:01 <ggainey> ow, right - accept and add, we def want the process to work 16:24:25 <bmbouter> oh this got added to sprint 16:24:30 <fao89> #idea Proposed for #8115: accept and add to sprint 16:24:30 <fao89> !propose other accept and add to sprint 16:24:31 <pulpbot> fao89: Proposed for #8115: accept and add to sprint 16:24:38 <fao89> #agreed accept and add to sprint 16:24:38 <fao89> !accept 16:24:38 <pulpbot> fao89: Current proposal accepted: accept and add to sprint 16:24:39 <bmbouter> so I think just triage here 16:24:39 <fao89> #topic https://pulp.plan.io/issues/8112 16:24:40 <pulpbot> fao89: 3 issues left to triage: 8112, 7950, 7922 16:24:41 <pulpbot> RM 8112 - quba42 - NEW - Update redmine job failed during pulp_deb 2.9.0 release pipeline. 16:24:42 <pulpbot> https://pulp.plan.io/issues/8112 16:25:15 <fao89> oh, this is release script issue 16:25:19 <fao89> #idea Proposed for #8112: Leave the issue as-is, accepting its current state. 16:25:19 <fao89> !propose accept 16:25:20 <pulpbot> fao89: Proposed for #8112: Leave the issue as-is, accepting its current state. 16:25:23 <ggainey> yeah 16:25:35 <ttereshc> should we add it to the epic? daviddavis 16:25:48 <daviddavis> sure, +1 16:25:51 * ttereshc is trying to find it 16:26:02 <fao89> not critical since it is the last step of the release and it has no issues to change status 16:26:17 <ttereshc> found it 16:26:23 <daviddavis> :) 16:26:24 <ttereshc> I'll add 16:26:35 <fao89> #agreed Leave the issue as-is, accepting its current state. 16:26:35 <fao89> !accept 16:26:35 <pulpbot> fao89: Current proposal accepted: Leave the issue as-is, accepting its current state. 16:26:36 <fao89> #topic https://pulp.plan.io/issues/7950 16:26:36 <pulpbot> fao89: 2 issues left to triage: 7950, 7922 16:26:37 <pulpbot> RM 7950 - newswangerd - NEW - Backport 7912 16:26:38 <pulpbot> https://pulp.plan.io/issues/7950 16:27:05 <ggainey> 7912 is still in POST 16:27:09 <ttereshc> still not merged 16:27:10 <ttereshc> yeah 16:27:12 <daviddavis> yea, skip 16:27:13 <fao89> !skip 16:27:14 <fao89> #topic https://pulp.plan.io/issues/7922 16:27:14 <pulpbot> fao89: 1 issues left to triage: 7922 16:27:15 <pulpbot> RM 7922 - bmclaugh - NEW - additional AVC denials 16:27:16 <pulpbot> https://pulp.plan.io/issues/7922 16:27:34 <bmbouter> this got put on hold from their side and it's not for released code 16:27:53 <bmbouter> and they've been notified of who they need to work w/ on the selinux team to have the changes made when they want to go forward with it 16:28:03 <bmbouter> so it's unreleased and not pulp's responsibility so I think CLOSE NOTABUG 16:28:08 <daviddavis> +1 16:28:10 <ttereshc> +1 16:28:11 <ggainey> sounds like a plan 16:28:12 <ggainey> +1 16:28:19 <bmbouter> they have a ticket tracking it 16:28:26 <fao89> #idea Proposed for #7922: close as not a bug 16:28:26 <fao89> !propose other close as not a bug 16:28:27 <pulpbot> fao89: Proposed for #7922: close as not a bug 16:28:31 <fao89> #agreed close as not a bug 16:28:31 <fao89> !accept 16:28:31 <pulpbot> fao89: Current proposal accepted: close as not a bug 16:28:32 <pulpbot> fao89: No issues to triage. 16:28:41 <ggainey> just in time :) 16:28:45 <daviddavis> buzzer beater with 2 min left 16:28:46 <ttereshc> we finished too early 16:28:47 <fao89> #endmeeting