14:33:02 <fao89> #startmeeting Pulp Triage 2020-04-14 14:33:02 <fao89> #info fao89 has joined triage 14:33:02 <fao89> !start 14:33:02 <pulpbot> Meeting started Tue Apr 14 14:33:02 2020 UTC. The chair is fao89. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:33:02 <pulpbot> Useful Commands: #action #agreed #help #info #idea #link #topic. 14:33:02 <pulpbot> The meeting name has been set to 'pulp_triage_2020-04-14' 14:33:02 <pulpbot> fao89: fao89 has joined triage 14:33:07 <ttereshc> !here 14:33:07 <ttereshc> #info ttereshc has joined triage 14:33:08 <pulpbot> ttereshc: ttereshc has joined triage 14:33:11 <fao89> !next 14:33:12 <pulpbot> fao89: 7 issues left to triage: 6497, 6481, 6474, 6469, 6468, 6465, 6463 14:33:12 <fao89> #topic https://pulp.plan.io/issues/6497 14:33:13 <pulpbot> RM 6497 - daviddavis - NEW - MasterModels have a TYPE attribute but it's not needed 14:33:14 <pulpbot> https://pulp.plan.io/issues/6497 14:33:15 <bmbouter> #info bmbouter has joined triage 14:33:15 <bmbouter> !here 14:33:15 <pulpbot> bmbouter: bmbouter has joined triage 14:33:26 <dkliban> #info dkliban has joined triage 14:33:26 <dkliban> !here 14:33:27 <pulpbot> dkliban: dkliban has joined triage 14:33:32 <bmbouter> I think accept and that's it for this 14:33:38 <ggainey> yeah same 14:33:39 <dkliban> yep 14:33:40 <fao89> #idea Proposed for #6497: Leave the issue as-is, accepting its current state. 14:33:40 <fao89> !propose accept 14:33:40 <pulpbot> fao89: Proposed for #6497: Leave the issue as-is, accepting its current state. 14:33:59 <ttereshc> +1 14:34:01 <fao89> #agreed Leave the issue as-is, accepting its current state. 14:34:01 <fao89> !accept 14:34:01 <pulpbot> fao89: Current proposal accepted: Leave the issue as-is, accepting its current state. 14:34:02 <pulpbot> fao89: 6 issues left to triage: 6481, 6474, 6469, 6468, 6465, 6463 14:34:02 <fao89> #topic https://pulp.plan.io/issues/6481 14:34:03 <pulpbot> RM 6481 - mdellweg - POST - test_delete_middle_version is sporadically failing 14:34:04 <pulpbot> https://pulp.plan.io/issues/6481 14:34:07 <ggainey> accept and add 14:34:14 <dkliban> yep ... i am working on this 14:34:27 <fao89> #idea Proposed for #6481: accept and add to sprint 14:34:27 <fao89> !propose other accept and add to sprint 14:34:27 <pulpbot> fao89: Proposed for #6481: accept and add to sprint 14:34:29 <ttereshc> +1 14:34:36 <fao89> #agreed accept and add to sprint 14:34:36 <fao89> !accept 14:34:36 <pulpbot> fao89: Current proposal accepted: accept and add to sprint 14:34:37 <fao89> #topic https://pulp.plan.io/issues/6474 14:34:37 <pulpbot> fao89: 5 issues left to triage: 6474, 6469, 6468, 6465, 6463 14:34:38 <pulpbot> RM 6474 - mdellweg - NEW - plugin_template generates a new directory that does not pass ci out of the box 14:34:39 <pulpbot> https://pulp.plan.io/issues/6474 14:34:57 <fao89> #idea Proposed for #6474: convert to a task 14:34:57 <fao89> !propose other convert to a task 14:34:57 <pulpbot> fao89: Proposed for #6474: convert to a task 14:34:58 <dkliban> x9c4: is there a PR for this already? 14:35:02 <bmbouter> I think accept and add to sprint 14:35:12 <dkliban> yeah ... accept and add to sprint 14:35:14 <bmbouter> we have to keep plugin_template working out of the box (to me) 14:35:21 <ggainey> same 14:35:36 <ttereshc> +1 14:35:38 <dkliban> i am pretty sure i was already reviewing a PR to fix this on friday 14:35:51 * dkliban didn't finish review 14:35:56 <fao89> I don't think he has a PR for that 14:36:10 <dkliban> anyway, let's accpe tand add to sprint 14:36:12 <fao89> #idea Proposed for #6474: accept and add to sprint 14:36:12 <fao89> !propose other accept and add to sprint 14:36:12 <pulpbot> fao89: Proposed for #6474: accept and add to sprint 14:36:18 <fao89> #agreed accept and add to sprint 14:36:18 <fao89> !accept 14:36:18 <pulpbot> fao89: Current proposal accepted: accept and add to sprint 14:36:19 <fao89> #topic https://pulp.plan.io/issues/6469 14:36:19 <pulpbot> fao89: 4 issues left to triage: 6469, 6468, 6465, 6463 14:36:20 <pulpbot> RM 6469 - dkliban@redhat.com - NEW - exceptions encountered during migration run are not captured in a task 14:36:21 <pulpbot> https://pulp.plan.io/issues/6469 14:36:52 <dkliban> ttereshc: ^ i filed that after our discussion ... shold we add to sprint? 14:36:53 <fao89> #idea Proposed for #6469: accept and add to sprint 14:36:53 <fao89> !propose other accept and add to sprint 14:36:53 <pulpbot> fao89: Proposed for #6469: accept and add to sprint 14:36:59 <dkliban> +1 14:37:00 <daviddavis> #info daviddavis has joined triage 14:37:00 <daviddavis> !here 14:37:00 <pulpbot> daviddavis: daviddavis has joined triage 14:37:13 <ttereshc> +1 and move to the migraiton plugin projetc 14:37:19 <fao89> #agreed accept and add to sprint 14:37:19 <fao89> !accept 14:37:19 <pulpbot> fao89: Current proposal accepted: accept and add to sprint 14:37:20 <fao89> #topic https://pulp.plan.io/issues/6468 14:37:20 <pulpbot> fao89: 3 issues left to triage: 6468, 6465, 6463 14:37:21 <mikedep333> #info mikedep333 has joined triage 14:37:21 <mikedep333> !here 14:37:22 <pulpbot> RM 6468 - daviddavis - NEW - Schema generation throws scary errors but succeeds 14:37:23 <pulpbot> https://pulp.plan.io/issues/6468 14:37:24 <pulpbot> mikedep333: mikedep333 has joined triage 14:37:50 <fao89> #idea Proposed for #6468: Leave the issue as-is, accepting its current state. 14:37:50 <fao89> !propose accept 14:37:50 <pulpbot> fao89: Proposed for #6468: Leave the issue as-is, accepting its current state. 14:37:58 <daviddavis> +1 14:38:04 <ggainey> yeah - we need to track this down but it's not immediate 14:38:13 <fao89> #agreed Leave the issue as-is, accepting its current state. 14:38:13 <fao89> !accept 14:38:13 <pulpbot> fao89: Current proposal accepted: Leave the issue as-is, accepting its current state. 14:38:14 <fao89> #topic https://pulp.plan.io/issues/6465 14:38:14 <pulpbot> fao89: 2 issues left to triage: 6465, 6463 14:38:15 <pulpbot> RM 6465 - dkliban@redhat.com - NEW - cancelling a 'complete' task returns strange result 14:38:16 <pulpbot> https://pulp.plan.io/issues/6465 14:38:17 <bmbouter> +1 14:38:28 <dkliban> i want to add tihs one to the sprint 14:38:33 <fao89> #idea Proposed for #6465: accept and add to sprint 14:38:33 <fao89> !propose other accept and add to sprint 14:38:33 <pulpbot> fao89: Proposed for #6465: accept and add to sprint 14:38:34 <daviddavis> works for me 14:38:36 <bmbouter> me too 14:38:40 <fao89> #agreed accept and add to sprint 14:38:40 <fao89> !accept 14:38:40 <pulpbot> fao89: Current proposal accepted: accept and add to sprint 14:38:40 <fao89> #topic https://pulp.plan.io/issues/6463 14:38:41 <pulpbot> fao89: 1 issues left to triage: 6463 14:38:41 <ttereshc> +1 14:38:42 <pulpbot> RM 6463 - binlinf0 - NEW - pulp 3.2.1 duplicate key error when sync 14:38:43 <pulpbot> https://pulp.plan.io/issues/6463 14:38:52 <dkliban> this one is tough to investigate 14:39:00 <dkliban> but i think we should add it to the sprint 14:39:06 <fao89> #idea Proposed for #6463: accept and add to sprint 14:39:06 <fao89> !propose other accept and add to sprint 14:39:06 <pulpbot> fao89: Proposed for #6463: accept and add to sprint 14:39:09 <ttereshc> yeah, we need a reproducer 14:39:16 <ggainey> or at least more info 14:39:21 <ggainey> not much there 14:39:30 <ttereshc> it's from the pulp-list, so we can ask 14:39:44 <fao89> #idea Proposed for #6463: skip 14:39:44 <fao89> !propose other skip 14:39:44 <pulpbot> fao89: Proposed for #6463: skip 14:40:00 <bmbouter> I asked him to file 14:40:04 <ggainey> yeah, there's not enough there to work on (imnsho, anyway) 14:40:15 <bmbouter> because I thought it was the same error that x9c4 14:40:16 <bmbouter> saw 14:40:31 <bmbouter> but I believe it was not the same error so asking for a clear reproducer is a good next step 14:40:38 <dkliban> let's skip 14:40:40 <bmbouter> I can ask and we can skip here if that's helpful 14:40:44 <fao89> #agreed skip 14:40:44 <fao89> !accept 14:40:44 <pulpbot> fao89: Current proposal accepted: skip 14:40:45 <pulpbot> fao89: No issues to triage. 14:40:51 <fao89> open floor! 14:41:33 <dkliban> i need ideas for investigating these intermittent test failures 14:41:43 <daviddavis> ha 14:41:49 <fao89> I want to bring attention to my PR: https://github.com/pulp/pulpcore/pull/645 it fixes pulp_ansible and pulp_file CI failures 14:42:29 <fao89> I still don't know exactly what the real problem is, but I wrote on my PR the steps I followed to find the "fix" 14:42:50 <bmbouter> daviddavis maybe you and I shoiuld look at that one together? 14:42:56 <bmbouter> anyone else is welcome also 14:42:59 <daviddavis> bmbouter: +1 14:43:01 <bmbouter> we looked last time iirc 14:43:12 <bmbouter> ty fao89 will look as soon as I can I put it on my list 14:43:21 <fao89> maybe it could be obvious for you, I'm not very familiar with django magic 14:43:35 <bmbouter> for these test failures here's my take: we have a serious and deep test problem it's not just one or two things 14:43:36 <daviddavis> fao89: just so I understand, this pr removes the ordering filter from all views and just sets it for named viewsets? 14:43:48 <fao89> yep 14:44:10 <bmbouter> I want to wait on the test convo until this one resolves. 645 questions/discussion is good 14:44:19 <daviddavis> for the test failures, I am still not sure I understand how we got here. we were fine a week or two ago. 14:44:25 <daviddavis> or am I mistaken? 14:44:33 <bmbouter> let's resolve 645 first 14:44:40 <daviddavis> ok 14:44:41 <bmbouter> 645 I think is a 3.3. release blocker 14:44:46 <daviddavis> agreed 14:44:46 <dkliban> +1 14:45:18 <bmbouter> ttereshc: if the release doesn't start until tomorrow due to ^ I'm ok w/ that. are you the releaser for 3.3.0? 14:46:02 <fao89> daviddavis, I'm not so sure, looking at pulpcore I would agree with you, but I haven't been tracking pulp_ansible and pulp_file so I'm not sure if we were fine 14:46:11 <bmbouter> also we need to pull back https://pulp.plan.io/issues/6347 from MODIFIED 14:46:21 <bmbouter> I'm doing that now (to ASSIGNED) 14:46:36 <bmbouter> POST actually 14:46:43 <daviddavis> +1 14:47:04 <bmbouter> and I set fao89 as the assignee since he contributed both PRs (ok that's done) 14:47:12 <ggainey> cool 14:48:51 <ttereshc> bmbouter, I am, I'm waiting on the outcome and wondering if it's purely test issue or pulp itself. If it's clear that we have sometimes only false negatives due to issues with tests, we can probably release, otherwise, I'd suggest to wait. WDYT? 14:49:22 <dkliban> ttereshc: there is more problems than just the tests 14:49:29 <fao89> taking pulp_ansible, it started to fail 5 days ago, and the #6347 PR was merge 6 days ago on pulpcore 14:49:31 <bmbouter> we have lots of test issues I've realized 14:49:39 <ttereshc> I see 14:49:43 <bmbouter> let's come back to the test issues 14:50:32 <bmbouter> here's my take we can't release 3.3.0 until so we would move it's release to tomorrow and upon resolution of 645 we should send an "ok to release" emial to the thread and ttereshc you could read that and know to start the release tomorrow 14:50:33 <bmbouter> wdyt? 14:50:34 <fao89> same thing with pulp_file 14:51:09 <fao89> so definitely ordering filter was breaking pulp_ansible and pulp_file since it was merged 14:51:21 <bmbouter> I agree based on the timeline you've identified 14:51:32 <dkliban> ttereshc: this is the blocker for 3.3.0 releease https://pulp.plan.io/issues/6347 14:52:23 <ttereshc> dkliban, I followed the discussion, thanks 14:53:09 <fao89> https://travis-ci.org/github/pulp/pulp_file/builds and https://travis-ci.org/github/pulp/pulp_ansible/builds and https://github.com/pulp/pulpcore/commit/c1d3fb37feb7b4dd7b65b8369bca65c1a5abfba0 14:53:42 <bmbouter> is there agreement in the plan or a counterpoposal? 7 min before I have to go to another meeting 14:55:25 <dkliban> bmbouter: that sounds fine to me 14:55:38 <bmbouter> ok that's a plan then 14:55:45 <bmbouter> here's the deal with the tests, we have major problems 14:56:03 <bmbouter> we've been having intermittent failures for a while but it was too small for us to notice 14:56:19 <dkliban> the biggest problem i see is that we have fixture repos that reuse file names 14:56:28 <bmbouter> I agree I believe that is our root cause 14:56:41 <fao89> +1 14:56:45 <bmbouter> and it's for this reason why we have to continue to use orphan cleanup in an incredible number of places 14:57:10 <bmbouter> and also if you start to really randomize the tests with pytest --random I think it would show even more failures (not verified but my belief) 14:57:46 <bmbouter> so the solution is to create a new process for all tests to follow where there are unique fixtures for each test 14:58:09 <bmbouter> and we then port all tests to use them ... this will be a large effort 14:58:51 <bmbouter> there are two options (and really only one to me) 14:59:15 <bmbouter> skip any test that is failing intermittently or stop releasing for maybe a month 14:59:32 <bmbouter> the latter I believe to be a no-go but put there for completeness 14:59:45 <fao89> I believe moving from unittest style to pytest would help 14:59:47 <dkliban> bmbouter: there is another option 14:59:56 <bmbouter> more options! bring it out 15:00:26 <dkliban> we can add a few more fixture repos manually and swap the intermittently failing tests to those for now 15:00:59 <fao89> I think we need a meeting for discussing the tests and define short, mid and long term goals 15:01:38 <bmbouter> I agree 15:01:41 <dkliban> +1 ... can we do it today? and who wants to participate? 15:01:45 <bmbouter> dkliban: that is a good middle ground option 15:01:47 <bmbouter> I want to 15:02:29 <ttereshc> I'd like to participate 15:02:37 <ttereshc> dkliban, ^ 15:03:15 <fao89> I also want to participate 15:03:44 <dkliban> can we meet in 30 mins? 1 hour? 15:04:12 <dkliban> i am availabl then 15:05:00 <fao89> galaxy meeting will finish in 1 hour 15:05:58 <ppicka> fao89: add me too please, I'd like to participate with tests. 15:06:15 <dkliban> i'll send an invite 15:06:18 <x9c4> I'd like to participate too. 15:06:28 <dkliban> fao89: we can end open floor now 15:06:30 <ttereshc> dkliban, I'm available whenever today 15:06:37 <fao89> #endmeeting 15:06:37 <fao89> !end