How to make good reproducible pandas examples

Having spent a decent amount of time watching both the r and pandas tags on SO, the impression that I get is that pandas questions are less likely to contain reproducible data. This is something that the R community has been pretty good about encouraging, and thanks to guides like this, newcomers are able to get some help on putting together these examples. People who are able to read these guides and come back with reproducible data will often have much better luck getting answers to their questions.

How can we create good reproducible examples for pandas questions? Simple dataframes can be put together, eg:

import pandas as pd
df = pd.DataFrame({'user': ['Bob', 'Jane', 'Alice'], 
                   'income': [40000, 50000, 42000]})

But many example datasets need more complicated structure, eg:

  • datetime indices or data
  • Multiple categorical variables (is there an equivalent to R's expand.grid() function, which produces all possible combinations of some given variables?)
  • MultiIndex or Panel data
  • For datasets that are hard to mock up using a few lines of code, is there an equivalent to R's dput() that allows you to generate copy-pasteable code to regenerate your datastructure?


    Note: The ideas here are pretty generic for StackOverflow, indeed questions.

    Disclaimer: Writing a good question is HARD.

    The Good:

  • do include small* example DataFrame, either as runnable code:

    In [1]: df = pd.DataFrame([[1, 2], [1, 3], [4, 6]], columns=['A', 'B'])
    

    or make it "copy and pasteable" using pd.read_clipboard(sep='ss+') , you can format the text for StackOverflow highlight and use Ctrl+K (or prepend four spaces to each line):

    In [2]: df
    Out[2]: 
       A  B
    0  1  2
    1  1  3
    2  4  6
    

    test pd.read_clipboard(sep='ss+') yourself.

    * I really do mean small , the vast majority of example DataFrames could be fewer than 6 rowscitation needed, and I bet I can do it in 5 rows. Can you reproduce the error with df = df.head() , if not fiddle around to see if you can make up a small DataFrame which exhibits the issue you are facing.

    * Every rule has an exception, the obvious one is for performance issues (in which case definitely use %timeit and possibly %prun), where you should generate (consider using np.random.seed so we have the exact same frame): df = pd.DataFrame(np.random.randn(100000000, 10)) . Saying that, "make this code fast for me" is not strictly on topic for the site...

  • write out the outcome you desire (similarly to above)

    In [3]: iwantthis
    Out[3]: 
       A  B
    0  1  5
    1  4  6
    

    Explain what the numbers come from: the 5 is sum of the B column for the rows where A is 1.

  • do show the code you've tried:

    In [4]: df.groupby('A').sum()
    Out[4]: 
       B
    A   
    1  5
    4  6
    

    But say what's incorrect: the A column is in the index rather than a column.

  • do show you've done some research (search the docs, search StackOverflow), give a summary:

    The docstring for sum simply states "Compute sum of group values"

    The groupby docs don't give any examples for this.

    Aside: the answer here is to use df.groupby('A', as_index=False).sum() .

  • if it's relevant that you have Timestamp columns, eg you're resampling or something, then be explicit and apply pd.to_datetime to them for good measure**.

    df['date'] = pd.to_datetime(df['date']) # this column ought to be date..
    

    ** Sometimes this is the issue itself: they were strings.

  • The Bad:

  • don't include a MultiIndex, which we can't copy and paste (see above), this is kind of a grievance with pandas default display but nonetheless annoying:

    In [11]: df
    Out[11]:
         C
    A B   
    1 2  3
      2  6
    

    The correct way is to include an ordinary DataFrame with a set_index call:

    In [12]: df = pd.DataFrame([[1, 2, 3], [1, 2, 6]], columns=['A', 'B', 'C']).set_index(['A', 'B'])
    
    In [13]: df
    Out[13]: 
         C
    A B   
    1 2  3
      2  6
    
  • do provide insight to what it is when giving the outcome you want:

       B
    A   
    1  1
    5  0
    

    Be specific about how you got the numbers (what are they)... double check they're correct.

  • If your code throws an error, do include the entire stacktrace (this can be edited out later if it's too noisy). Show the line number (and the corresponding line of your code which it's raising against).

  • The Ugly:

  • don't link to a csv we don't have access to (ideally don't link to an external source at all...)

    df = pd.read_csv('my_secret_file.csv')  # ideally with lots of parsing options
    

    Most data is proprietary we get that: Make up similar data and see if you can reproduce the problem (something small).

  • don't explain the situation vaguely in words, like you have a DataFrame which is "large", mention some of the column names in passing (be sure not to mention their dtypes). Try and go into lots of detail about something which is completely meaningless without seeing the actual context. Presumably noone is even going to read to the end of this paragraph.

    Essays are bad, it's easier with small examples.

  • don't include 10+ (100+??) lines of data munging before getting to your actual question.

    Please, we see enough of this in our day jobs. We want to help, but not like this....
    Cut the intro, and just show the relevant DataFrames (or small versions of them) in the step which is causing you trouble.

  • Anyways, have fun learning python, numpy and pandas!


    How to create sample datasets

    This is to mainly to expand on @AndyHayden's answer by providing examples of how you can create sample dataframes. Pandas and (especially) numpy give you a variety of tools for this such that you can generally create a reasonable facsimile of any real dataset with just a few lines of code.

    After importing numpy and pandas, be sure to provide a random seed if you want folks to be able to exactly reproduce your data and results.

    import numpy as np
    import pandas as pd
    
    np.random.seed(123)
    

    A kitchen sink example

    Here's an example showing a variety of things you can do. All kinds of useful sample dataframes could be created from a subset of this:

    df = pd.DataFrame({ 
    
        # some ways to create random data
        'a':np.random.randn(6),
        'b':np.random.choice( [5,7,np.nan], 6),
        'c':np.random.choice( ['panda','python','shark'], 6),
    
        # some ways to create systematic groups for indexing or groupby
        # this is similar to r's expand.grid(), see note 2 below
        'd':np.repeat( range(3), 2 ),
        'e':np.tile(   range(2), 3 ),
    
        # a date range and set of random dates
        'f':pd.date_range('1/1/2011', periods=6, freq='D'),
        'g':np.random.choice( pd.date_range('1/1/2011', periods=365, 
                              freq='D'), 6, replace=False) 
        })
    

    This produces:

              a   b       c  d  e          f          g
    0 -1.085631 NaN   panda  0  0 2011-01-01 2011-08-12
    1  0.997345   7   shark  0  1 2011-01-02 2011-11-10
    2  0.282978   5   panda  1  0 2011-01-03 2011-10-30
    3 -1.506295   7  python  1  1 2011-01-04 2011-09-07
    4 -0.578600 NaN   shark  2  0 2011-01-05 2011-02-27
    5  1.651437   7  python  2  1 2011-01-06 2011-02-03
    

    Some notes:

  • np.repeat and np.tile (columns d and e ) are very useful for creating groups and indices in a very regular way. For 2 columns, this can be used to easily duplicate r's expand.grid() but is also more flexible in ability to provide a subset of all permutations. However, for 3 or more columns the syntax quickly becomes unwieldy.
  • For a more direct replacement for r's expand.grid() see the itertools solution in the pandas cookbook or the np.meshgrid solution shown here. Those will allow any number of dimensions.
  • You can do quite a bit with np.random.choice . For example, in column g , we have a random selection of 6 dates from 2011. Additionally, by setting replace=False we can assure these dates are unique -- very handy if we want to use this as an index with unique values.
  • Fake stock market data

    In addition to taking subsets of the above code, you can further combine the techniques to do just about anything. For example, here's a short example that combines np.tile and date_range to create sample ticker data for 4 stocks covering the same dates:

    stocks = pd.DataFrame({ 
        'ticker':np.repeat( ['aapl','goog','yhoo','msft'], 25 ),
        'date':np.tile( pd.date_range('1/1/2011', periods=25, freq='D'), 4 ),
        'price':(np.random.randn(100).cumsum() + 10) })
    

    Now we have a sample dataset with 100 lines (25 dates per ticker), but we have only used 4 lines to do it, making it easy for everyone else to reproduce without copying and pasting 100 lines of code. You can then display subsets of the data if it helps to explain your question:

    >>> stocks.head(5)
    
            date      price ticker
    0 2011-01-01   9.497412   aapl
    1 2011-01-02  10.261908   aapl
    2 2011-01-03   9.438538   aapl
    3 2011-01-04   9.515958   aapl
    4 2011-01-05   7.554070   aapl
    
    >>> stocks.groupby('ticker').head(2)
    
             date      price ticker
    0  2011-01-01   9.497412   aapl
    1  2011-01-02  10.261908   aapl
    25 2011-01-01   8.277772   goog
    26 2011-01-02   7.714916   goog
    50 2011-01-01   5.613023   yhoo
    51 2011-01-02   6.397686   yhoo
    75 2011-01-01  11.736584   msft
    76 2011-01-02  11.944519   msft
    

    Diary of an Answerer

    My best advice for asking questions would be to play on the psychology of the people who answer questions. Being one of those people, I can give insight into why I answer certain questions and why I don't answer others.

    Motivations

    I'm motivated to answer questions for several reasons

  • Stackoverflow.com has been a tremendously valuable resource to me. I wanted to give back.
  • In my efforts to give back, I've found this site to be an even more powerful resource than before. Answering questions is a learning experience for me and I like to learn. Read this answer and comment from another vet. This kind of interaction makes me happy.
  • I like points!
  • See #3.
  • I like interesting problems.
  • All my purest intentions are great and all, but I get that satisfaction if I answer 1 question or 30. What drives my choices for which questions to answer has a huge component of point maximization.

    I'll also spend time on interesting problems but that is few and far between and doesn't help an asker who needs a solution to a non-interesting question. Your best bet to get me to answer a question is to serve that question up on a platter ripe for me to answer it with as little effort as possible. If I'm looking at two questions and one has code I can copy paste to create all the variables I need... I'm taking that one! I'll come back to the other one if I have time, maybe.

    Main Advice

    Make it easy for the people answering questions.

  • Provide code that creates variables that are needed.
  • Minimize that code. If my eyes glaze over as I look at the post, I'm on to the next question or getting back to whatever else I'm doing.
  • Think about what you're asking and be specific. We want to see what you've done because natural languages (English) are inexact and confusing. Code samples of what you've tried help resolve inconsistencies in a natural language description.
  • PLEASE show what you expect!!! I have to sit down and try things. I almost never know the answer to a question without trying some things out. If I don't see an example of what you're looking for, I might pass on the question because I don't feel like guessing.
  • Your reputation is more than just your reputation.

    I like points (I mentioned that above). But those points aren't really really my reputation. My real reputation is an amalgamation of what others on the site think of me. I strive to be fair and honest and I hope others can see that. What that means for an asker is, we remember the behaviors of askers. If you don't select answers and upvote good answers, I remember. If you behave in ways I don't like or in ways I do like, I remember. This also plays into which questions I'll answer.


    Anyway, I can probably go on, but I'll spare all of you who actually read this.

    链接地址: http://www.djcxy.com/p/24886.html

    上一篇: (这是什么巫术?

    下一篇: 如何制作好重现熊猫的例子