Preliminary Research on Research

ChengXiang Zhai

Department of Computer Science Graduate School of Library & Information Science Institute for Genomic Biology, Statistics University of Illinois, Urbana-Champaign

http://www-faculty.cs.uiuc.edu/~czhai, [email protected]

2014 ADC PhD School in Big Data, The University of Queensland, Brisbane, Australia, July 13, 2014 1

What is Research?

• • • • Research – –

Discover new knowledge Seek answers to questions

Basic research –

Goal: Expand man’s knowledge (e.g., which genes control social behavior of honey bees? )

– –

Often driven by curiosity (but not always) High impact examples: relativity theory, DNA, …

Applied research –

Goal: Improve human condition (i.e., improve the wolrd) (e.g., how to cure cancers?)

– –

Driven by practical needs High impact examples: computers, transistors, vaccinations, …

The boundary is vague; distinction isn’t important 2

Why Research?

Curiosity

Amount of knowledge Advancement of Technology Utility of Applications Quality of Life Basic Research Applied Research Application Development 3

Where’s Your Position?

Different position benefits from different collaborators

Quality of Life Amount of knowledge Advancement of Technology Utility of Applications Basic Research Applied Research Application Development 4

Thoughts on Impact

• • Impact (Q, A)=ImportanceOfProblem(Q)*QualityofSolution(A) –

QualityOfAnswer(A): [0,100/100], thus

–

Impact(Q,A) <=ImportanceOfProblem(Q)

• Interactive impact: –

Make Impact(P1, …,Pk) >=Impact(P1)+…+Impact(Pk)

–

This is possible if P1, …,Pk form a coherent story (together they enable something bigger)

Suggestions: –

Think big



maximize ImportanceOfProblem

–

Solve synergistic problems



maximize interactive impact

–

Strive for BEST solution



maximize QualityofSolution

5

Discovery-Impact Tradeoff

Disocvery of knowleedge (# citations) Goal Impact on society (# users; $$$) 6

What is a Good Research Problem?

• Well defined: Would we be able to tell whether we’ve solved the problem?

• Highly important: Who would care about the solution to the problem? What would happen if we don’t solve the problem?

• Solvable: Is there any clue about how to solve it? Do you have a baseline approach? Do you have the needed resources?

• Matching your strength: Are you at a good position to solve the problem? 7

How to Find a Problem?

• Application-driven (Find a nail, then make a hammer) –

Identify a need by people/users that cannot be satisfied well currently (“complaints” about current data/information management systems?)

–

How difficult is it to solve the problem?

• No big technical challenges: do a startup • Lots of big challenges: write a research proposal –

Identify one technical challenge as your topic

–

Formulate/frame the problem appropriately so that you can solve it

• Aim at a completely new application/function (find a high stake nail) 8

How to Find a Problem?

(cont.)

• Tool-driven (Hold a hammer, and look for a nail) –

Choose your favorite state-of-the-art tools

• Ideally, you have a “secret weapon” • Otherwise, bring tools from area X to area Y – – –

Look around for possible applications Find a novel application that seems to match your tools How difficult is it to use your tools to solve the problem?

• • No big technical challenges: do a startup Lots of big challenges: write a research proposal – –

Identify one technical challenge as your topic Formulate/frame the problem appropriately so that you can solve it

• Aim at important extension of the tool (find an unexpected application and use the best hammer) 9

How to Find a Problem?

(cont.)

• In practice, you do both in various kinds of ways –

You talk to people in application domains and identify new “nails”

–

You take courses and read books to acquire new “hammers”

–

You check out related areas for both new “nails” and new “hammers”

–

You read visionary papers and the “future work” sections of research papers, and then take a problem from there

–

…

10

• • • • • •

General Steps to Define a Research Problem

Generate and Test Raise a question Novelty test: Figure out to what extent we know how to answer the question – –

There’s already an answer to it: Is the answer good enough?

• • Yes: not interesting, but can you make the question more challenging? No: your research problem is how to get a better answer to the raised question

No obvious answer: you’ve got an interesting problem to work on

Tractability test: Figure out whether the raised question can be answered – –

I can see a way to answer it or potentially answer it: you’ve got a solvable problem I can’t easily see a way to answer it: Is it because the question is too hard or you’ve not worked hard enough? Try to reframe the problem to make it easier

Evaluation test: Can you obtain a data set and define measures to test solutions/answers? – –

Yes: you’ve got a clearly defined problem to work on No: can you think of anyway to indirectly test the solutions/answers? Can you reframe the problem to fit the data?

Every time you reframe a problem, try to do all the three tests again.

11

What It Takes to Do Research

• • • • • • • Curiosity: allow you to ask questions Critical thinking: allow you to challenge assumptions Learning: take you to the frontier of knowledge Persistence: so that you don’t give up Respect data and truth: ensure your research is solid Communication: allow you to publish your work … 12

Critical Thinking

• • • • Develop a habit of asking questions, especially why questions Always try to make sense of what you have read/heard; don’t let any question pass by Get used to challenging everything Practical advice –

Question every claim made in a paper or a talk (can you argue the other way?)

–

Try to write two opposite reviews of a paper (one mainly to argue for accepting the paper and the other for rejecting it)

–

Force yourself to challenge one point in every talk that you attend and raise a question

13

Respect Data and Truth

• Be honest with the experiment results –

Don’t throw away negative results!

–

Try to learn from negative results

• Don’t twist data to fit your hypothesis; instead, let the hypothesis choose data • Be objective in data analysis and interpretation; don’t mislead readers • • Aim at understanding/explanation instead of just good results Be careful not to over-generalize (for both good and bad results); you may be far from the truth 14

Communications

• General communication skills: –

Oral and written

–

Formal and informal

–

Talk to people with different level of backgrounds

• Be clear, concise, accurate, and adaptive (elaborate with examples, summarize by abstraction) • • English proficiency Get used to talking to people from different fields 15

Persistence

• • • Work only on topics that you are passionate about Work only on hypotheses that you believe in Don’t draw negative conclusions prematurely and give up easily –

positive results may be hidden in negative results

–

In many cases, negative results don’t completely reject a hypothesis

• Be comfortable with criticisms about your work (learn from negative reviews of a rejected paper) • Think of possibilities of repositioning a work 16

Optimize Your Training

• Know your strengths and weaknesses –

strong in math vs. strong in system development

–

creative vs. thorough

–

…

• • • Train yourself to fix weaknesses Find strategic partners Position yourself to take advantage of your strengths 17

Optimizing “Research Return”: Pick a Problem Best for You

Your Passion High (Potential) Impact Your Strength Best problems for you Find your passion: If you can choose any job in the world, what would you do? Test of impact: If you are given $1M to fund a research project, what would you fund? Find your strength/Avoid your weakness: What are you (not) good at? 18

• •

Typical Structure of a Research Paper

1. Introduction – – – –

Background discussion to motivate your problem Define your problem Argue why it’s important to solve the problem Identify knowledge gap in existing work or point out deficiency of existing answers/solutions

– –

Summarize your contributions Briefly mention potential impact

Tips: – –

Start with sentences understandable to almost everyone Tell the story at a high-level so that the entire introduction is understandable to people with no/little technical background in the topic

–

Use examples if possible

19

• •

Typical Structure of a Research Paper (cont.)

2. Previous/Related work – –

Sometimes this part is included in the introduction or appears later Previous work = work that you extend (readers must be familiar with it to understand your contribution)

–

Related work = work related to your work (readers can until later in the paper to know about it)

Tips: –

Make sure not to miss important related work

– –

Always safer to include more related work Discuss the existing work and its connection to your work

• • • Your work extends … Your work is similar to … but differs in that … Your work represents an alternative way of … –

Whenever possible, explicitly discuss your contribution in the context of existing work

20

• •

Typical Structure of a Research Paper (cont.)

3. Problem definition/formulation –

Clearly define your problem

• If it’s a new problem, discuss its relation to existing related problems • If it’s an old problem, cite the previous work –

Justify why you define the problem in this way

–

Discuss challenges in solving the problem

Tips: –

Give both an informal description and a formal description if possible

–

Make sure that you mention any assumption you make when defining the problem (e.g., your focus may be on studying the problem in certain conditions)

21

• • •

Typical Structure of a Research Paper (cont.)

4. Overview of the solution(s) (can be merged with the next part) –

Give a high-level information description of the proposed solutions or solutions you study

–

Use examples if possible

5. Specific components of your solution(s) – –

Be precise (formal description helps) Use intuitive descriptions to help people understand it

Tips: – –

make sure that you organize this part so that it’s understandable to people with various backgrounds Don’t just throw in formulas; include high-level intuitive descriptions whenever possible

22

•

Typical Structure of a Research Paper (cont.)

6. Experiment design: make sure you justify it –

Data set

–

Measures

–

Experiment procedure

• Tips: –

Given enough details so that people can reproduce your experiments

–

Discuss limitation/bias if any, and discuss its potential influence on your study

23

• •

Typical Structure of a Research Paper (cont.)

7. Result analysis: –

Organized based on research questions to be answered or hypotheses tested

– –

Be comprehensive, but focus on the major conclusions Include “standard” components

• • • • • Baseline comparison Individual component analysis Parameter sensitivity analysis Individual query analysis Significance test –

Discuss the influence of any bias or limitation

Tips –

Don’t leave any question unanswered (try to provide an explanation for all the observed results)

–

Discuss your findings in the context of existing work if possible

• • Similar observations have also been made in … This is in contrast to … observed in … One explanation is …. 24

•

Typical Structure of a Research Paper (cont.)

8. Conclusions and future work –

Summarize your contributions

–

Discuss its potential impact

–

Discuss its limitation and point out directions for future work

• 9. References 25

Tips on Polishing your Paper

• • • • Start with the core messages you want to convey in the paper and expand your paper by following the core story • Try to convey the core messages at different levels so that people with different knowledge background can all get them Try to write a review of your paper yourself, commenting on its originality, technical soundness, significance, evaluation, etc, and then revise the paper if needed Check out reviewer’s instructions, e.g., the following: http://nips07.stanford.edu/nips07reviewers.html

(not necessarily matching your conference, but should share a lot of common requirements) Try to polish English as much as you can 26

After You Get Reviews Back

• • • Carefully classify comments into: –

Unreasonable comments (e.g., misunderstanding):

• Try to improve the clarity of your writing –

Reasonable comments

• • Constructive: easy to implement Non-constructive: think about it, either argue the other way or mention weakness of your work in the paper If paper is accepted – –

Take the last chance to polish the paper as much as you can You’ll regret if later you discover an inaccurate statement or a typo in your published paper

If paper is rejected – – – –

Digest comments and try to improve the research work and the paper Run more experiments if necessary Don’t try to please reviewers (the next reviewer might say something opposite); instead use your own judgments and use their comments to help improve your judgments Reposition the paper if necessary (again, don’t reposition it just because a reviewer rejected your original positioning)

27

Summary

• Research is about discovery and increase our knowledge (innovation & understanding) • Intellectual curiosity and critical thinking are extremely important • • Work on important problems that you are passionate about Aim at becoming a top expert on one topic area –

Obtain complete knowledge about the literature on the topic (read all the important papers and monitor the progress)

–

Write a survey if appropriate

–

Publish one or more high-quality papers on the topic

• Don’t give up! 28