15:30:08 <fao89> #startmeeting Pulp Triage 2021-01-22
15:30:08 <fao89> !start
15:30:08 <fao89> #info fao89 has joined triage
15:30:08 <pulpbot> Meeting started Fri Jan 22 15:30:08 2021 UTC.  The chair is fao89. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:30:08 <pulpbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:30:08 <pulpbot> The meeting name has been set to 'pulp_triage_2021-01-22'
15:30:08 <pulpbot> fao89: fao89 has joined triage
15:30:29 <fao89> Open Floor!
15:30:35 <daviddavis> #info daviddavis has joined triage
15:30:35 <daviddavis> !here
15:30:35 <pulpbot> daviddavis: daviddavis has joined triage
15:30:37 <ppicka> #info ppicka has joined triage
15:30:37 <ppicka> !here
15:30:37 <pulpbot> ppicka: ppicka has joined triage
15:30:40 <ggainey> #info ggainey has joined triage
15:30:40 <ggainey> !here
15:30:40 <pulpbot> ggainey: ggainey has joined triage
15:31:03 <fao89> topic: Spam script removing actual users
15:31:12 <bmbouter> #info bmbouter has joined triage
15:31:12 <bmbouter> !here
15:31:12 <pulpbot> bmbouter: bmbouter has joined triage
15:31:28 <daviddavis> where were we with this on tuesday
15:31:59 <fao89> I don't remember, but it was a pretty intense conversation
15:32:03 <daviddavis> lol
15:32:14 <daviddavis> we were talking about making potential spam content private
15:32:34 <bmbouter> oh yeah and I was saying that goes against a long standing goal motivated by many problems
15:32:36 <dkliban> #info dkliban has joined triage
15:32:36 <dkliban> !here
15:32:36 <pulpbot> dkliban: dkliban has joined triage
15:32:37 <gerrod> #info gerrod has joined triage
15:32:37 <gerrod> !here
15:32:37 <pulpbot> gerrod: gerrod has joined triage
15:32:44 <fao89> oh yeah, and bmbouter was thinking about not allowing private content
15:32:49 <ttereshc> #info ttereshc has joined triage
15:32:49 <ttereshc> !here
15:32:49 <pulpbot> ttereshc: ttereshc has joined triage
15:33:02 <ggainey> heh - yeah, I think we'd gotten to "we don't remove users anymore, we should prob be making content private, but we want to get rid of the private-button altogether"
15:33:04 <dkliban> yeah ... i was going to file a task to make the script mark things private ... but i didn't do that
15:33:58 <bmbouter> yeah I don't want to trade one problem for a new problem
15:34:05 <bmbouter> so what is the root cause of our current problem?
15:34:08 <x9c4> #info x9c4 has joined triage
15:34:08 <x9c4> !here
15:34:08 <pulpbot> x9c4: x9c4 has joined triage
15:34:17 <daviddavis> valid content is being removed as spam
15:34:29 <ggainey> "spam filtering is hard and can have false positives"
15:34:57 <fao89> I think we are just trusting on CI code from pulp-ci, spam and bugzilla jobs
15:35:34 <fao89> we deployed the code and we are just praying for it to be right
15:36:15 <gerrod> is there still spam on GH issues? aren't we looking to switch anyway?
15:37:54 <ggainey> if we're worried about flase-positives, we could just turn off the spam-code, and everyone take onboard a atask of "check faily activity dashboard on plan.io and hand-remove obvious spam"
15:38:02 <ggainey> (which is what we were doing before)
15:38:09 <ggainey> ^flase^false
15:38:38 <ggainey> esp if we're thinking of migrating to using GH "soon"(ish)
15:38:45 <fao89> I'm seeing less spam, maybe because of the script? Can we disable it and see if we are having less spam even without the script?
15:38:52 <bmbouter> we are looking to switch
15:38:54 <ttereshc> gerrod, we are planning to, just we need to do something before we've moved
15:38:57 <x9c4> Can we change the script to flag spam?
15:38:58 <ggainey> sounds like a useful experiment
15:39:10 <fao89> https://github.com/pulp/pulp-ci/actions
15:39:13 <daviddavis> how do we flag spam?
15:39:20 <ggainey> (turning it off for a bit to see how bad things actually are right now, that is)
15:39:40 <x9c4> We add a label and then it is easier to review what needs to be deleted.
15:39:44 <bmbouter> redmine doesn't have a flag facility and adding more custom fields takes us away from the goal of switching to GHI
15:39:46 <dalley> the spam still happens even with it turned on, isn't it only deleted after a week or something
15:39:59 <ggainey> we run once/wk iirc
15:40:25 <bmbouter> the question I keep asking is: is this a severe enough issue for us to take an action
15:40:46 <bmbouter> my conclusion to that atm is no even if someusers content got deleted because in fairness they didn't follow the process they were emailed about
15:40:51 <daviddavis> hard to tell because we don't know how much content has actually been removed
15:41:16 <daviddavis> if we thinking about doing nothing, I would maybe just disable the spam script
15:41:28 <ggainey> daviddavis: at some point, "sever" is defined by "how many ppl are yelling at us" :)
15:42:27 <ggainey> but I'd be OK w/disabling the script and seeing how bad the problem is right now - as long as we as a team agree that looking for and removing obvious spam "by hand' is something we'll do semi-regularly
15:42:43 <fao89> https://github.com/pulp/pulp-ci/blob/master/.github/spam_cleaner.py
15:43:08 <daviddavis> one of my concerns is that no one is monitoring or maintaining this script
15:43:20 <fao89> +1
15:43:27 <bmbouter> this is also my main concer
15:43:29 <bmbouter> concern
15:43:47 <daviddavis> if someone were, I'd be less concerned about keeping it running
15:44:23 <fao89> we have a code that interact with users and we don't monitor it, so I vote to disable it
15:44:51 <bmbouter> that makes sense to me also
15:45:11 <bmbouter> also the spam isn't killing us but having someone monitor spam takes away from productive work
15:45:13 <ggainey> I'm good with it - it was a good experiment, but see prev RE "spam filtering is hard"
15:45:23 <dkliban> yep
15:45:46 <dkliban> should we go ahead and disable it today?
15:45:50 <ggainey> so let's turn it off and see if we're overwhelmed - may not be a problem, the spam-ecosystem is always changing
15:45:53 <ggainey> yes
15:46:03 <ggainey> or rather, "I am in favor" :)
15:46:12 <x9c4> +1
15:46:24 <bmbouter> +1
15:46:28 <daviddavis> +1
15:47:05 <dkliban> +1
15:47:14 <dkliban> i'll make a PR to disable it now
15:47:27 <fao89> dkliban++
15:47:27 <pulpbot> fao89: dkliban's karma is now 581
15:47:29 * ggainey cheers wildly (but not spammily)
15:47:32 <bmbouter> ty
15:47:37 <dkliban> lol
15:47:39 <fao89> next topic: New tags in Redmine
15:47:40 <daviddavis> excellentayyyy
15:48:37 <fao89> who can speak bout the new tags? Is it just FYI?
15:48:44 <daviddavis> I believe dalley added it
15:49:00 <x9c4> bmbouter, will you bring up tags to delete again if we add new ones?
15:49:21 <bmbouter> like when we switch to GHI (github issues?)
15:49:27 <bmbouter> if so, no because those tags will become labels
15:49:39 <bmbouter> I still don't think we're ready to capture pulp.next things though
15:49:51 <bmbouter> we did this in pulp2 and spent effort on planning pulp3 only to get into pulp3 and throw all of it out
15:50:31 <ttereshc> bmbouter, this came up at backlog grooming
15:50:53 <ttereshc> there are some really good issues/stories,very well written but we know we won't get to them\
15:51:23 <ttereshc> they are up to date so far
15:51:42 <ttereshc> I agree with not planning pulp.next too early
15:51:46 <bmbouter> maybe keeping this is exactly what we should do
15:52:04 <bmbouter> but we were just saying that the longer the time between planning and implementation the more that planning becomes wasted
15:52:09 <fao89> Pulp-NG
15:52:17 <ttereshc> bmbouter,  I agree
15:52:36 <bmbouter> and we may loose what we already have
15:52:38 <ttereshc> I'm on the fence, just a pity to throw away some of them
15:52:49 <bmbouter> but by keeping it I'm concerned we'll continue to build it out and throw good time after bad
15:52:55 <bmbouter> throw away
15:53:08 <bmbouter> fyi it's not a big deal for me, this is just my opinion
15:53:09 <daviddavis> I still don't understand the hesitation with closing issues personally. it's not like we're deleting them
15:53:20 <bmbouter> I feel the same
15:53:48 <ttereshc> how many times have you re-opened the story or issue instead of wrting a new one?
15:53:49 <ggainey> if we close 'pulp next' issues with a standard comment, we can find them all via search, yeah?
15:54:25 <daviddavis> probably the same number of times I've filed a dupe issue for an already open one
15:54:27 <ttereshc> I don't have strong opinion here, just sharing a sentiment I noticed at backlog grooming
15:54:52 <ttereshc> dalley, any thoughts? I believe you are in favor of not closing
15:54:55 <daviddavis> yea, I don't have a strong opinion either
15:57:29 <daviddavis> maybe return to this when dalley is present?
15:57:50 <bmbouter> +1
15:58:20 <ttereshc> +1
15:58:30 <ggainey> +1
15:58:35 <fao89> next topic: Make worker ttl configurable
15:58:50 <dalley> as long as we tag them so that we can find them later to re-open them, I'm OK with closing them
15:59:10 <bmbouter> we could make a custom query w/ the issue numbers
15:59:11 <dalley> I'm against closing important / useful features or issues without a way to dig them back up
15:59:30 <bmbouter> that way we don't take on another thing to migrate to GHI and yet we don't loose them
16:00:01 <bmbouter> just an idea
16:01:01 <ttereshc> I can speak to the next topic
16:01:35 <bmbouter> I can make a query to demo it and we can leave the pulp.next issues open for now
16:01:48 <bmbouter> if someone can put on the next open floor to revisit
16:01:59 <fao89> I put it
16:02:13 <bmbouter> that'll work let's go into next yes?
16:02:22 <ggainey> +1
16:02:32 <daviddavis> +1
16:02:37 <dalley> +1
16:02:41 <fao89> last topic: Make worker ttl configurable
16:02:49 <ttereshc> Make the hearbeat for workers configurable is a katello ask and it's because of the issue with io when customers have hdd's.
16:03:03 <bmbouter> ohhhhh
16:03:15 <ttereshc> and it's noticeable when running the migration plugin.
16:03:31 <bmbouter> yeah so this comes up maybe once a month and I usually say it's because we're doing something wrong elsewhere and the db is blocking in an unhealthy way
16:03:37 <ttereshc> it's likely that ther are things to improve in plugin itself
16:03:56 <ttereshc> but it's quite easilty reproducible if you switch from ssd to hdd
16:04:00 <bmbouter> if we're in a situation where a single write can't occur for 30 seconds we have a bigger problem
16:04:41 <ttereshc> bmbouter, current dogfood server doesn't use very modern hardware and the hearbeat for pulp2 was increased there for that reason
16:05:10 <bmbouter> yes but this is signalling a deeper problem
16:05:12 <ttereshc> folks say that it reflects a situation which some customers are often in
16:05:19 <bmbouter> I agree they are in that situation
16:05:33 <bmbouter> but I'm trying to shift the convo to investigating the root cause not the symptom
16:05:36 <dalley> the problem is that there's just a huge amount of IO on a slow disk and no way to prioritize certain activities over others
16:05:55 <bmbouter> then we shouldn't be driving the db that hard or have a backpressure monitor
16:06:05 <bmbouter> consider that other services run on that postgres server too
16:06:12 <ttereshc> so we are looking into improving migration plugin and track what causes that, we already fixed a lot
16:06:12 <dalley> bmbouter, it's not just the DB, it's file ops too
16:06:31 <bmbouter> if we're DoSing the db and the filesystem we are DoSing more than just pulp's heartbeats
16:06:54 <ttereshc> I think those can go in parallel, make it configurable and figure out deeper problems and if it's possible to fix them
16:07:31 <bmbouter> I'm not confident we will address the real problem if we do that
16:07:42 <ttereshc> if we do what?
16:07:49 <ttereshc> go into both directions?
16:07:54 <bmbouter> yes
16:08:04 <bmbouter> I'm uncomfortable with that option at least
16:08:23 <ttereshc> maybe we should have a convo with katello folks on pulp-dev
16:08:41 <bmbouter> I'm into that, I really really want to make it usable for them
16:08:45 <ggainey> +1
16:08:57 <ttereshc> it's for customers on 3.18
16:09:14 <ttereshc> pulpcore 3.7
16:09:27 <bmbouter> yup and we've known about this for a long time
16:09:50 <ttereshc> I think we can move on to the triage and I'll bring it up on pulp-dev, wdyt?
16:09:58 <bmbouter> that sounds great
16:10:02 <daviddavis> +1
16:10:19 <fao89> !next
16:10:21 <fao89> #topic https://pulp.plan.io/issues/8131
16:10:21 <pulpbot> fao89: 7 issues left to triage: 8131, 8122, 8116, 8115, 8112, 7950, 7922
16:10:22 <pulpbot> RM 8131 - mcorr - POST - Add links to pulp plugin docs into the body of docs.pulpproject.org
16:10:23 <pulpbot> https://pulp.plan.io/issues/8131
16:10:50 <fao89> #idea Proposed for #8131: convert to task
16:10:50 <fao89> !propose other convert to task
16:10:50 <pulpbot> fao89: Proposed for #8131: convert to task
16:11:04 <daviddavis> +1
16:11:05 <ggainey> +1
16:11:09 <bmbouter> +1
16:11:10 <ppicka> +1
16:11:11 <fao89> #agreed convert to task
16:11:11 <fao89> !accept
16:11:11 <pulpbot> fao89: Current proposal accepted: convert to task
16:11:11 <ttereshc> it's in POST
16:11:12 <fao89> #topic https://pulp.plan.io/issues/8122
16:11:12 <pulpbot> fao89: 6 issues left to triage: 8122, 8116, 8115, 8112, 7950, 7922
16:11:13 <pulpbot> RM 8122 - iballou - NEW -  Backport Request: 8099 (File upload causes django.security.SuspiciousFileOperation:ERROR) to 3.9
16:11:14 <pulpbot> https://pulp.plan.io/issues/8122
16:11:17 <ttereshc> are we adding such to the sprint?
16:12:51 <bmbouter> for 8131 I'm ok w/ that
16:12:58 <bmbouter> mcorr do you plan to take as assigned? ^
16:13:44 <mcorr> I raised a PR for that bmbouter  - I am sorry for making trouble
16:13:54 <bmbouter> mcorr: no trouble!
16:14:02 <bmbouter> got a link?
16:14:11 <mcorr> sure one sec
16:14:28 <fao89> https://github.com/pulp/pulpcore/pull/1079
16:14:38 <mcorr> https://github.com/pulp/pulpcore/pull/1079 bmbouter
16:14:42 <mcorr> aah thanks fao89
16:14:52 <fao89> pulpbot did add the link to the issue
16:14:52 <pulpbot> fao89: Error: "did" is not a valid command.
16:15:00 <bmbouter> so yeah add to sprint plz
16:15:09 <fao89> about the backport should I accept and add?
16:15:22 <daviddavis> I think it's done already
16:15:22 <bmbouter> I think so but dkliban would know ^?
16:15:23 <daviddavis> dkliban: ?
16:15:25 <bmbouter> yup
16:15:33 <ttereshc> I think it's not a backport, since thre is no 3.10
16:15:44 <daviddavis> it says to 3.9
16:16:02 <ttereshc> yeah, so it's a normal Z stream, no?
16:16:07 <fao89> yeah, it is just z-stream
16:16:16 <ttereshc> there is no release to backport it from
16:16:25 <daviddavis> I thought any issue to a z-stream was a backport
16:16:57 <ttereshc> I think we can have a fix with a db migration in a z stream
16:17:01 <ttereshc> if it's a latest one
16:17:09 <ttereshc> but we can't backport such
16:17:24 <daviddavis> I don't think we need a db migration for https://pulp.plan.io/issues/8122
16:17:43 <ttereshc> I'm just saying in theory
16:17:56 <ttereshc> the diff between backport and the latest z stream
16:18:14 <ttereshc> anyway, I'm +1 to close 8122
16:18:17 <ttereshc> as a dupe
16:18:19 <daviddavis> so to me, backport means taking something from master or a later release and applying to a release branch and releasing
16:19:06 <fao89> #idea Proposed for #8122: close and point the link to z-stream
16:19:06 <fao89> !propose other close and point the link to z-stream
16:19:06 <pulpbot> fao89: Proposed for #8122: close and point the link to z-stream
16:19:12 <ggainey> +1
16:20:01 <fao89> #agreed close and point the link to z-stream
16:20:01 <fao89> !accept
16:20:01 <pulpbot> fao89: Current proposal accepted: close and point the link to z-stream
16:20:02 <fao89> #topic https://pulp.plan.io/issues/8116
16:20:02 <pulpbot> fao89: 5 issues left to triage: 8116, 8115, 8112, 7950, 7922
16:20:03 <pulpbot> RM 8116 - paji@redhat.com - NEW - Export is generating the incorrect 'next_version' causing import to fail.
16:20:04 <pulpbot> https://pulp.plan.io/issues/8116
16:20:05 <dkliban> re 8122: it should be closed
16:20:07 <ttereshc> daviddavis, my understanding excludes the master branch  if it goes to the latest Z, but I can see your logic as well
16:20:23 <ggainey> 8116 - accept-and-add please - we need to fix this
16:20:37 <fao89> #idea Proposed for #8116: accept and add to sprint
16:20:37 <fao89> !propose other accept and add to sprint
16:20:37 <pulpbot> fao89: Proposed for #8116: accept and add to sprint
16:20:59 <daviddavis> I'm not sure how the workflow is to create these other-things-which-are-not-backports. do users still file backports?
16:21:33 <ggainey> gang, can we move the bakports-discussion to some othe rtime? we have a lot of issues to get thru
16:21:37 <ttereshc> they just ask to release a z stream I think and include that fix
16:21:41 <fao89> I think we can add backport to the next open floor
16:21:45 <ggainey> +1
16:21:57 <daviddavis> yea we discuss this more
16:22:04 <ttereshc> whatever works for everyone I'm fine with
16:22:57 <fao89> 8116 anyone agains accepting and adding to sprint?
16:23:08 <daviddavis> nope, works for me
16:23:14 <bmbouter> wfm
16:23:19 <fao89> #agreed accept and add to sprint
16:23:19 <fao89> !accept
16:23:21 <pulpbot> fao89: Current proposal accepted: accept and add to sprint
16:23:22 <fao89> #topic https://pulp.plan.io/issues/8115
16:23:22 <pulpbot> fao89: 4 issues left to triage: 8115, 8112, 7950, 7922
16:23:23 <pulpbot> RM 8115 - bmbouter - NEW - Satellite issue sync is broken
16:23:24 <pulpbot> https://pulp.plan.io/issues/8115
16:24:01 <ggainey> ow, right - accept and add, we def want the process to work
16:24:25 <bmbouter> oh this got added to sprint
16:24:30 <fao89> #idea Proposed for #8115: accept and add to sprint
16:24:30 <fao89> !propose other accept and add to sprint
16:24:31 <pulpbot> fao89: Proposed for #8115: accept and add to sprint
16:24:38 <fao89> #agreed accept and add to sprint
16:24:38 <fao89> !accept
16:24:38 <pulpbot> fao89: Current proposal accepted: accept and add to sprint
16:24:39 <bmbouter> so I think just triage here
16:24:39 <fao89> #topic https://pulp.plan.io/issues/8112
16:24:40 <pulpbot> fao89: 3 issues left to triage: 8112, 7950, 7922
16:24:41 <pulpbot> RM 8112 - quba42 - NEW - Update redmine job failed during pulp_deb 2.9.0 release pipeline.
16:24:42 <pulpbot> https://pulp.plan.io/issues/8112
16:25:15 <fao89> oh, this is release script issue
16:25:19 <fao89> #idea Proposed for #8112: Leave the issue as-is, accepting its current state.
16:25:19 <fao89> !propose accept
16:25:20 <pulpbot> fao89: Proposed for #8112: Leave the issue as-is, accepting its current state.
16:25:23 <ggainey> yeah
16:25:35 <ttereshc> should we add it to the epic? daviddavis
16:25:48 <daviddavis> sure, +1
16:25:51 * ttereshc is trying to find it
16:26:02 <fao89> not critical since it is the last step of the release and it has no issues to change status
16:26:17 <ttereshc> found it
16:26:23 <daviddavis> :)
16:26:24 <ttereshc> I'll add
16:26:35 <fao89> #agreed Leave the issue as-is, accepting its current state.
16:26:35 <fao89> !accept
16:26:35 <pulpbot> fao89: Current proposal accepted: Leave the issue as-is, accepting its current state.
16:26:36 <fao89> #topic https://pulp.plan.io/issues/7950
16:26:36 <pulpbot> fao89: 2 issues left to triage: 7950, 7922
16:26:37 <pulpbot> RM 7950 - newswangerd - NEW - Backport 7912
16:26:38 <pulpbot> https://pulp.plan.io/issues/7950
16:27:05 <ggainey> 7912 is still in POST
16:27:09 <ttereshc> still not merged
16:27:10 <ttereshc> yeah
16:27:12 <daviddavis> yea, skip
16:27:13 <fao89> !skip
16:27:14 <fao89> #topic https://pulp.plan.io/issues/7922
16:27:14 <pulpbot> fao89: 1 issues left to triage: 7922
16:27:15 <pulpbot> RM 7922 - bmclaugh - NEW - additional AVC denials
16:27:16 <pulpbot> https://pulp.plan.io/issues/7922
16:27:34 <bmbouter> this got put on hold from their side and it's not for released code
16:27:53 <bmbouter> and they've been notified of who they need to work w/ on the selinux team to have the changes made when they want to go forward with it
16:28:03 <bmbouter> so it's unreleased and not pulp's responsibility so I think CLOSE NOTABUG
16:28:08 <daviddavis> +1
16:28:10 <ttereshc> +1
16:28:11 <ggainey> sounds like a plan
16:28:12 <ggainey> +1
16:28:19 <bmbouter> they have a ticket tracking it
16:28:26 <fao89> #idea Proposed for #7922: close as not a bug
16:28:26 <fao89> !propose other close as not a bug
16:28:27 <pulpbot> fao89: Proposed for #7922: close as not a bug
16:28:31 <fao89> #agreed close as not a bug
16:28:31 <fao89> !accept
16:28:31 <pulpbot> fao89: Current proposal accepted: close as not a bug
16:28:32 <pulpbot> fao89: No issues to triage.
16:28:41 <ggainey> just in time :)
16:28:45 <daviddavis> buzzer beater with 2 min left
16:28:46 <ttereshc> we finished too early
16:28:47 <fao89> #endmeeting
16:28:47 <fao89> !end