Many research students in Australia will be planning to submit their thesis next month. Let’s fast forward to that sweet moment you find out your examiners reports are back, or completed your Viva and been told you have passed. Congratulations! Time to ring up the bank and the passport office to get that long awaited ‘Dr’ in front of your name right?
You cannot use the title of Doctor until you graduate.
To graduate you must first get your thesis published in the university library. This involves submitting a ‘camera ready’ or complete copy of your thesis to the relevant administrators, who will then pass it to the librarians to catalogue. I’m writing this post because Laura left a message on the feedback forum asking questions and I realised I have never got around to talking about this aspect of the process.
When I started Whispering, about seven years ago, we used to lodge bound copies in the university library stacks. Now most universities have a public, online repository. This has been a great development; increased accessibility means increased relevance. But before your thesis can take its place there, you must attend to the changes that your examiners have suggested that you make to your thesis.
(If you are in the USA, you may want to stop reading now unless you are interested in what happens in other countries. What I have to say pertains to the Australian and UK system, and some other parts of the world, usually those who were formally colonies of the UK)
How many amendments you have to make will depend on how your thesis was categorised. In most universities I have worked or studied in there is a 5 point categorising system, with varying amounts of time allowed to make the amendments as follows:
- Category One: no changes, around 2 weeks to submit final camera ready document to the online university repository
- Category Two*: minor amendments, usually 6 weeks to submit
- Category Three*: major amendments, mostly 3 months to submit
- Category Four: revise and resubmit for examination, 1 year
- Category Five: fail
*Primary supervisor has to approve the final version.
The categories I listed above are not a scoring system, despite what the numbers might suggest. A PhD is a pass or fail proposition; the scoring system is just a way of communicating how much change is needed before you can pass. In Australia you will get category Four before you get category Five, so you’ll have at least one year to pull yourself out of the hole if the examiners do not think you did a good enough job. Only 2% of research students, nationally, fail on their second attempt. To put this in perspective, as my old boss Denise Cuthbert used to say, with an average attrition rate of around 30% nationally, there’s far more risk of never completing than failure.
The examination process is full of ambiguity. One examiner might think problems with how you have numbered the footnotes means you should get a category two, but another might merely tell you to fix it and give you a category one. There’s no need to panic if you get category 1, 2 or 3 – your thesis can still be considered of high quality. I got category two for my PhD, but both examiners checked the ‘outstanding work’ box which made me feel better. I later won the faculty award for best thesis and was one of the runners up for the university prize (dammit!), so category two didn’t hold me back a bit.
Everyone who has amendments will find it disheartening, if not demoralising. But remember that the overwhelming majority of people have to do amendments of some sort. At RMIT, where I used to work, around 89% of people had to do changes suggested by the examiner before they could submit their camera ready document. Here are five suggestions if you get a Category Two or Three and have to negotiate changes with your supervisory team (I’ll write another post about category four because I think the issues are much more extreme for those who have to be re-examined).
1) Do it now, and as quickly as you can
You’ve probably waited up to 5 months for your exam results or the chance to do your Viva. By the time the reports come in life has moved on; you may have a new job or even be living in a new country. It can be hard to even open that document you sweated over and a time limit of a couple of months can seem both daunting and depressing. But just because you have a couple of months, doesn’t mean it will necessarily take that long. I’ve known people to work through category three amendments in less than a week. The category two changes on my PhD took a day of hard work. The trick is to do the bare minimum of the suggested changes, which leads me to my next point.
2) Work out which amendments you really have to make and which ones you will refuse to do
No one’s work is perfect, so swallow your pride and try to read the reports as objectively as you can – how many of these changes do you HAVE to make? The examiner is not the expert – you are; the report is a list of suggestions, not a shopping list. So take control and address only those concerns you think are important.
In a perfect world you will have a civilised meeting with your supervisors to discuss a plan of attack before you make substantive changes to your document. I should caution you, however, that there’s potential for conflict here. Some supervisors, particularly inexperienced ones, are under the impression that their job is to make sure you carry out every suggestion, no matter how ridiculous. Make yourself familiar with the regulations around examination so you can explain them if necessary.
3) Always make a cogent and well argued case for not taking up a suggestion from the examiners
You’ll need good reasons why you will not make a suggested amendment. Some reasons are more acceptable than others and the language you use is important. Never complain it will take too long; state how and why the changes suggested are impractical within the timeframe you have been given. I think you should resist, as far as you are able, any requests to collect more data. A thesis that honestly needs more data was poorly designed in the first place and should never have passed. If data is highlighted as an issue, suggest to your supervisor that you do more analysis or interpretation instead.
4) Summarise the changes as you go
Your supervisors will want to assess the changes you’ve made without reading through the whole thesis again. This is why it’s important to document the changes as you go in the form of a rejoinder. You might use a table with three columns ‘suggestion’, ‘response’ and ‘page numbers’, itemising each change as you go. Alternatively you can arrange the suggested changes in themes, and make a written statement of the changes under each heading. This is what I did in attached PDF which is a response to Examiner comments on my own PhD. As you will see, it’s a formal document similar to the kind of document you write for journal editors when you have made changes to your article, but perhaps a bit more forceful.
5) Cross the T’s and dot the I’s
Usually you will submit a PDF to the repository, which might present some challenges to those who have video or other data which accompanies the written work. Check the instructions and get help from the library if you need to. Many universities require you to get copyright permission for any images or tables used in your thesis that you did not make yourself. This involves locating the original copyright holder and writing to them.
Getting copyright permission can take anywhere from 24 hours to never; sometimes even locating the original copyright holder can be tricky. I’ll do a post on this process in the near future, but I mention it here to highlight that there are a lot of small details you need to attend to before your thesis will be considered acceptable. I’ve known people to spend months, even up to a year, working on these details – such a waste of time.
I hope this cheat sheet helps you in the final stages. Are you about to complete and have questions about this process? I might not be able to answer all of them, but feel free to ask away in the comments. I’m hoping some of the experienced supervisors, whom I know read the blog, might chime in and help with advice and suggestions in addition to what I have written here.
Finally – best of luck with your submission!
4 things you should know about choosing your examiners
5 rookie researcher mistakes
Ph.D. Defense Post Mortem and Advice for Others
NOTE 1: This is part 1 in a series that will probably contain 3 or 4 parts. Then I will return to the usual data science etc. posts.
NOTE 2: This post was intentionally delayed until I received final approval on the submission of my final dissertation.
On March 14, I passed my final oral defense for my Ph.D. in Statistics.
It was the moment I had been waiting for. The moment I dreaded for so many years and the moment that I thought would never come. I had very few days and times to choose from. It came down to the 13th (bad luck?) and the 14th (Pi Day). My defense began at 10am and was supposed to take a max of 2 hours. I left home around 7am to be sure I arrived in enough time after what could be a long ride. I arrived at 8:45am after sitting in rush hour traffic for almost 2 hours. I printed color copies of my slides and four copies of my dissertation and set up each committee member’s spot nicely: one water, slides, dissertation, a plate and a napkin. I placed a big tray of color-sprinkled cookies in the middle of the table equidistant from each member’s “assigned” seat. The IT manager set up my laptop so it would project to the portable projector, and set up a second laptop which would serve as one of my committee members — my original advisor retired and was participating remotely via Skype.
Then the waiting started. At 9:50am my original advisor rang in on Skype and gave me a quick wave while chewing on a toothpick, a signature quirk. At 9:59am, nobody else was there and I started to panic. A few minutes later I got an email that one member was sitting outside a locked door on the other side of the hallway. I went to meet him, and the rest of my committee all arrived together, including my advisor, who I was warned by others is always late. I was then kicked out of the room while the committee deliberated my case, decided who would ask what questions, and what behavioral tests may be used (I am convinced this is a thing in PhD defenses, and I suppose it’s useful). Then a problem arose. Nobody could hear my remote member. I ran down the hallway to get the IT manager and he brought in some speakers that really didn’t help. Sigh I thought. We had no choice but to continue.
My advisor had joked that I had chosen a very “nice” committee. My advisor himself is very soft spoken, but knows how to deliver constructive criticism. He has this down to an art. I had never known this about him before working with him in this capacity. Another committee member is on the younger end and is perhaps even more soft-spoken than my advisor. I had heard that he is very rigorous in terms of theory and has high expectations in that regard. Many of his comments trailed off and I couldn’t really hear them. My outside member is known for being very friendly and I always felt this way about him (but I had forgotten something — more on that later). And nobody could hear my previous advisor, which was a shame. The day before my defense, my advisor was trying to pep me up speaking very highly of my work and that this should be pretty easy if not a “slam dunk.” So I was feeling pretty good, though he warned me “X really appreciates theory, so it would be helpful if you can frame your research theoretically.” The color left my face as I thought to myself “you’re telling me this now?” I had become accustomed to minor surprises though. During each of our meetings, my advisor would at one point put his clenched knuckles to his lower lip and stare deep into space for about 60 seconds or so, and return with a suggestion that required significant thought. It was almost like he was running through all combinations and permutations of how things could go down in an effort to be “preemptive” (that’s the word he used).
I won’t go into a narrative of everything that happened during the defense because that would take all day. It was a tough experience, but in hindsight it was a positive experience. Actually, it is a good thing I am writing this now because the committee’s feedback makes a lot more sense now and made my dissertation stronger. I consider myself open to constructive criticism (and even flat out criticism) but I was so caught off-guard and confused I sort of wish I could have seen the look on my face.
Fast forward to the aftershow:
My advisor was smiling, but there was no “congratulations” which scared me. He invited me into the room, closed the door, and we all quickly discussed some small changes. I took notes, and soon everyone left. On their way out they congratulated me on passing the defense.
After all that, nobody ate any of the cookies or drank the water.
My advisor said that my defense went “very smoothly,” and was “gentle” though it felt “brutal.” I admit it could have been a ton worse. These are some of the comments I received during my defense, which kind of made me think I had failed, but I had to keep up a facade that I was confident about what I was doing:
- My dissertation title, and the name I gave to the method I developed, was rejected.
- “The mathematical notation is very critical, and doesn’t seem dissertation-quality.”
- Thought in my head: “I am meticulous about notation. Are you serious?” His arguments made sense though which I will discuss later.
- Thought in my head: “So are 99% of the methods in the literature!”
- Thought in my head: “I had a very easy to understand baseline, and my method exceeded that baseline.”
- Thought in my head: “Please no. How am I going to relate computer vision (a field that kind of scares me) methods to NLP?” and “But, neural networks are not my strong suit…”
- Thought in my flustered head: “WTF is a truncated SVD? This has nothing do with with truncated or censored data!” He then mentioned the k in the subscript in my SVD and immediately “Oh, yeah, it’s a truncated SVD”
- I won’t comment on the thought in my head. 😉
Most of the feedback above came from two of my members, with my advisor basically agreeing with them which made me nervous, but I found out why later. Sometimes if you can agree with people on the small stuff, they won’t be so eager to move on to harder things. I’ve been told that I am humble, so I know that my job as a student and professional in a setting like this is to shut up and listen because these people know much more than I do about what I am doing in terms of theory. I am actually much more comfortable in a technical-explained-practically setting. Anyway, I had seven strategies to get through the defense:
- Be confident. You did a lot of work and you know this work better than anyone else. Besides, there is a quote that the dissertation is “supposed to be the worst research you ever write, and is the beginning of your research career, not the end.”
- Be cooperative. (My advisor actually told me this one, but it’s obvious). The faculty want you to pass (my committee was very friendly, not all are), and they want you to also write a good manuscript.
- Be humble and patient. It’s possible someone on my committee will be in the dark (possibly the outside member) and/or you may assume the committee is more familiar with your specialization (i.e. machine learning, finance, whatever) than they are.
- Defend yourself, and push back (hard) if needed. Everybody of course has an opinion, but each should be respected (especially in this setting). My goal was to acknowledge and appreciate their suggestion, but push back if I felt someone’s suggestion was infeasible, was confusing, or did not make sense for my work. (YMMV here though as I think I gave my advisor the wrong idea though but that’s always bound to happen depending on personality…)
- Readily admit “I Don’t Know.” I actually never had to use this, but considering the wringer I was put through in terms of questioning, I would have had no problem saying “I don’t know” if I truly did not know the answer to the question.
- Take tons of notes to show you care about their feedback (you should!).
- While organizing my research and writing the dissertation, I questioned myself about everything. “Why did you choose a topic model instead of a regular classifier?” “What kind of cross-validation did you do and why?” “What is the point of this? Why is this method not stupid?” “Why didn’t you use Deep Learning?” Except for Deep Learning, none of the questions on my “list” were asked.
I had a few big hiccups that I could have avoided:
- My advisor suggested that I meet with my committee with my slides and have them go through them with me. Some of the issues I faced (notation) could have been fixed much earlier. Also, the slides were not in the sequence that one of the committee members was expecting. It lead to the awkward “what problem are you solving?” question. I had this exact same problem in my preliminary oral. If your advisor tells you to do something, do it, even it sounds like a “mere suggestion.”
- Somehow, I must have given the wrong impression that I took the feedback personally. The thing is, I can be soft-spoken and intimidated when talking with faculty, but I am not that way in general, so I think they may have been surprised when I was a little more assertive than I usually was with them. Also, I figured out that my advisor may have interpreted the word “brutal” literally when all I meant was that it was “tough.” Because of that, he thought I was offended. Later on, he had used another word quite literally and I figured out what was going on.
- It had been years since I had witnessed a final oral defense, and it was in a different department (Computer Science), not my own (Statistics). I do recall the one Stats defense I attended was very tense (even compared to CS) and I genuinely thought the guy was going to fail. The committee sounded like they hated his research, one member was late and the student called him out on it, and at one point the candidate told one of the members (paraphrased) “all of your questions could have been answered by actually reading my dissertation when I sent it to you a month ago” and his advisor laughed. He passed. And everyone was smiles and laughter afterwards. Go see a dissertation defense in your own department before you defend!
- Not really a hiccup, but it had been a while since I had seen my outside member in a research setting. At the last seminar of his I presented at, I discussed a fun project I was working on, something I did not intend to publish. I remember he was very vocal and asked me really tough questions about my work that I was not expecting at all. This is what faculty is supposed to do. I wish I had taken that project more seriously and looked at it through publishing rather than engineering curiosity. I had forgotten about that experience, but boy did the memories come back during the defense.
My advisor met with me for about 20 minutes after my defense to go over some minor revisions and he spent quite a bit of time explaining why the committee had the feedback they did. I think he realized how surprised I was at the amount of “tension” in the room. I did not take any of it personally because a defense is not supposed to be easy, but since he was much more critical of my work compared to the past ten weeks, it threw me for a loop. We discussed the changes to be made and explained that this is like submitting a paper to a journal. If you want the paper to be published and a referee says to do something, just do it, even if it may not make sense. I agree with him and after thinking about their feedback more, I realize that there is a kernel of truth in each suggestion, and part of this process is learning how to incorporate feedback into something that makes sense:
- The title was confusing and suggested that I was researching something very theoretical, and it seems the committee wasn’t satisfied enough theory had gone into the work assuming the title was correct as it was. I had been lazy with my terminology, and I changed the title to something more believable.
- The notation was on par with some research papers I have read, but I remember feeling so confused about their notation. I had made a lot of assumptions and placed a lot of conditions on my sampling method, and those conditions were not reflected in the notation. So, this was a fair point and I ended up redoing all of the notation (a frustrating process) and adding a “notation guide” in the appendix.
- The ad-hoc nature ended up being fine. The feedback came from the theoretical member and our program graduates dissertations that are applied in nature.
- Not proving the method works was the hardest one to agree with but even as I was doing the research, I felt like something was missing in my comparison because it was my method of creating the baseline. I quickly did another experiment with someone else’s baseline.
- Data augmentation was an area I had completely missed in my literature review. I avoided it because it seemed to only used with neural networks and computer vision. I instead focused on “query expansion” which is still used to this day, but the theory behind it is quite dated. I found through my expanded literature review that Data Augmentation was the other side of the coin and filled in some of the gaps in my thought process. I saw other researchers that had done similar (though still quite different) things that I did, but were much more relatable than query expansion. Honestly, it got me a bit more interested in computer vision and neural networks because it suggested that there were some principled methods and not just magic black boxes.
I took about a week off and then started making my edits. I used several different color pens to mark where I needed to make changes and what those changes should be. After all was said and done, that draft looked like it had open heart surgery performed on it. It seemed like no word went untouched.
Anyway, I hope this advice can help someone. “You’ll get through it” is what I say with some trepidation because it is an experience I do not want to go through again!
For anyone interested, my defense slides are here and my final dissertation is here.